Light-emitting user input device for calibration or pairing

ABSTRACT

A light emitting user input device can include a touch sensitive portion configured to accept user input (e.g., from a user&#39;s thumb) and a light emitting portion configured to output a light pattern. The light pattern can be used to assist the user in interacting with the user input device. Examples include emulating a multi-degree-of-freedom controller, indicating scrolling or swiping actions, indicating presence of objects nearby the device, indicating receipt of notifications, assisting pairing the user input device with another device, or assisting calibrating the user input device. The light emitting user input device can be used to provide user input to a wearable device, such as, e.g., a head mounted display device.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.15/965,702, filed Apr. 27, 2018, entitled “LIGHT-EMITTING USER INPUTDEVICE FOR CALIBRATION OR PAIRING,” which claims the benefit of priorityunder 35 U.S.C. § 119(e) to U.S. Provisional Application No. 62/490,863,filed on Apr. 27, 2017, entitled “LIGHT-EMITTING USER INPUT DEVICE,” thedisclosure of which is hereby incorporated by reference herein in itsentirety.

FIELD

The present disclosure relates to virtual reality and augmented realityimaging and visualization systems and more particularly to a lightemitting user input device associated with the imaging and visualizationsystems.

BACKGROUND

Modern computing and display technologies have facilitated thedevelopment of systems for so called “virtual reality” “augmentedreality” or “mixed reality” experiences, wherein digitally reproducedimages or portions thereof are presented to a user in a manner whereinthey seem to be, or may be perceived as, real. A virtual reality, or“VR”, scenario typically involves presentation of digital or virtualimage information without transparency to other actual real-world visualinput; an augmented reality, or “AR”, scenario typically involvespresentation of digital or virtual image information as an augmentationto visualization of the actual world around the user; a mixed reality,or “MR”, related to merging real and virtual worlds to produce newenvironments where physical and virtual objects co-exist and interact inreal time. As it turns out, the human visual perception system is verycomplex, and producing a VR, AR, or MR technology that facilitates acomfortable, natural-feeling, rich presentation of virtual imageelements amongst other virtual or real-world imagery elements ischallenging. Systems and methods disclosed herein address variouschallenges related to VR, AR and MR technology.

SUMMARY

Examples of a light-emitting user input device are disclosed.Embodiments of the user input device can be used to provide input to anAR, VR, or MR device. The light-emitting user input device can alsoprovide visual information of events or objects associated with theAR/VR/MR device to the user or people in the user's environment.

The light emitting user input device can include a touch sensitiveportion configured to accept a user input (e.g., from a user's thumb)and a light emitting portion configured to output a light pattern. Thelight pattern can be used to assist the user in interacting with theuser input device. Examples include emulating a multi-degree-of-freedomcontroller, indicating scrolling or swiping actions, indicating presenceof objects nearby the device, indicating receipt of notifications,assisting calibrating the user input device, or assisting pairing theuser input device with another device. The light emitting user inputdevice can be used to provide user input to a head mounted displaysystem such as, e.g., a mixed reality display device.

Details of one or more implementations of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages will becomeapparent from the description, the drawings, and the claims. Neitherthis summary nor the following detailed description purports to defineor limit the scope of the inventive subject matter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson.

FIG. 2 schematically illustrates an example of a wearable system.

FIG. 3 schematically illustrates aspects of an approach for simulatingthree-dimensional imagery using multiple depth planes.

FIG. 4 schematically illustrates an example of a waveguide stack foroutputting image information to a user.

FIG. 5 shows example exit beams that may be outputted by a waveguide.

FIG. 6 is a schematic diagram showing an optical system including awaveguide apparatus, an optical coupler subsystem to optically couplelight to or from the waveguide apparatus, and a control subsystem, usedin the generation of a multi-focal volumetric display, image, or lightfield.

FIG. 7 is a block diagram of an example of a wearable system.

FIG. 8 is a process flow diagram of an example of a method of renderingvirtual content in relation to recognized objects.

FIG. 9 is a block diagram of another example of a wearable system.

FIG. 10 is a process flow diagram of an example of a method fordetermining user input to a wearable system.

FIG. 11 is a process flow diagram of an example of a method forinteracting with a virtual user interface.

FIG. 12A illustrates side and front (user-facing) views of an example ofa totem.

FIG. 12B illustrates a top view of another example of a totem.

FIG. 13A illustrates a cross-sectional view of an example touchpad of atotem.

FIG. 13B illustrates examples of touch screen technologies.

FIGS. 13C and 13D illustrate additional cross-sectional views of theexample touchpad of a totem.

FIG. 13E illustrates a bottom view of the example touchpad.

FIG. 14A illustrates a top view of an example touchpad of a totem.

FIG. 14B illustrates an overview of an example layout of LEDs associatedwith the touchpad.

FIG. 15 illustrates example LED layouts or patterns of light from an LEDlayout.

FIGS. 16A and 16B illustrate example placement or movement patterns oflight emissions of a totem.

FIG. 17A is a block diagram that illustrates example components of atotem.

FIG. 17B is a side cross-section view illustrating components of anotherexample totem.

FIGS. 18A-18D illustrate an example programming interface forconfiguring placement or movement patterns for light emissions from ahalo of a totem.

FIGS. 19A-19C illustrate examples of totem calibration using a halo oflight emissions from the totem.

FIGS. 19D and 19E illustrate an example of totem calibration using lightpatterns associated with a halo.

FIGS. 20A and 20B illustrate examples of indicating a wireless pairingprocess between a mixed reality device and a totem with a halo.

FIG. 20C illustrates an example process of device pairing with a halo.

FIG. 20D illustrates another example process of device pairing with ahalo.

FIG. 21A illustrates an example of indicating the status of the totem.

FIG. 21B illustrates an example of light placement or movement patternsduring a power on and off process.

FIG. 21C illustrates an example of light placement or movement patternswhich show a battery charging status.

FIG. 21D illustrates an example light pattern when the totem has enteredinto a sleep mode.

FIG. 21E illustrates an example process of indicating a status of atotem based on light placement or movement patterns.

FIGS. 22A and 22B illustrate example light placement or movementpatterns that are used as cues for user interactions.

FIG. 22C illustrates another example of using a light pattern to providean indication of an available user interface operation.

FIG. 23 illustrates an example of using a light pattern as an alert toindicate an incorrect or improper user interaction.

FIG. 24A illustrates an example light pattern for a swipe gesture.

FIG. 24B illustrates an example light pattern for a touch gesture.

FIG. 24C illustrates an example process of providing a cue for userinteractions on a totem.

FIG. 25A illustrates an example interactive use of the light guide.

FIG. 25B illustrates an example interactive use of a totem with twointeractable regions.

FIG. 25C illustrates an example process for interacting with a totem.

FIGS. 26A and 26B illustrate examples of interacting with physicalobjects using a totem.

FIG. 27 illustrates an example of moving a virtual object with a sixdegrees-of-freedom (6DOF) totem.

FIGS. 28A and 28B illustrate examples of providing information ofobjects via placement and movement of light patterns.

FIG. 28C illustrates an example process for providing informationassociated with an object using light patterns.

FIG. 29A illustrates example light placement or movement patternsindicating the receipt of a notification.

FIG. 29B illustrates an example process for providing a notificationusing light patterns on a totem.

FIG. 30 illustrates an example light pattern that can be used to informa person in the user's environment of the user's current interaction.

Throughout the drawings, reference numbers may be re-used to indicatecorrespondence between referenced elements. The drawings are provided toillustrate example embodiments described herein and are not intended tolimit the scope of the disclosure.

DETAILED DESCRIPTION

Overview of a Light-Emitting User Input Device

A touch sensitive user input device can support user inputs such asswiping, tapping, clicking, pressing, etc. For example, when a user usesa touchpad to browse a website, the user can use one finger (e.g., thethumb) to swipe left and right to move the webpage left and right or tapup and down to move the webpage up and down. In order to achieve moreuser interface functionalities, such as, e.g., snapping content to adesired location, scrolling, or resizing the content, the touchsensitive user input device often requires multiple figures. Forexample, the user may use two fingers to enlarge the webpage and use onefinger to move within the webpage. When the touchpad is part of ahandheld user input device for a wearable device (which may includeAR/VR/MR wearable display), however, the user may not have as manyfingers available to interact with the touchpad. For example, the usermay use the thumb to interact with the touchpad while use other fingersto hold the totem. As a result, the types of user interfacefunctionalities that can be achieved by the touchpad of conventionalhandheld devices may significantly decrease.

To ameliorate this problem, the touchpad of embodiments of the totemdescribed herein can be divided into multiple interactable regions whereeach region may be mapped to one or more types of user interfaceinteractions. For example, the touchpad may include a touch surfacewhich is near the center of the touchpad and an outer region at leastpartially surrounding the touch surface. The outer region may include alight guide configured to output light patterns (e.g., the placement,illumination, color and/or movement of the light) that assist the userin interacting with the totem. The light patterns outputted by the lightguide may sometimes be referred to herein as a “halo,” because the lightpatterns may appear to surround a central, touch-sensitive portion ofthe totem. The light guide may be on top of a touch sensor such that theuser can interact with the light guide and provide touch sensitive inputto the totem via the light guide region. When the user actuates thetouch surface, the totem may simulate cursor actions (such as, e.g.,moving forward and back on a browser). However, if a user wants toperform other types user interface operations (such as, e.g., scrollinga webpage), the user may actuate the light guide. In some examples, thelight guide may not be touch-sensitive. The user can actuate a regionnear the light guide (e.g., a region on the touch pad which may besurrounded by the light guide) to actuate the totem.

As another example, the touch surface can be divided into twointeractable regions with one region supporting the user's touch actions(such as e.g., simulating the functions of a multiple degree of freedom(DOF) directional d-pad) while another region supporting the user'sswiping actions. The touch surface with two interactable regions mayinclude concentric rings with one inner ring (as the first interactableregion) and one outer ring (as the second interactable region). In thisexample, the light guide surrounding the touch surface may or may not beinteractable but it can provide the visual feedback related to user'sinteractions or related to the wearable system. As will further bedescribed herein, the type of user interactions supported by aninteractable region may be changed dynamically based on events in thevirtual environment or objects that the user is interacting with. Forexample, the outer region may be used as a d-pad when a user is browsinga web whereas the same region may support a swipe interaction (e.g., acirculate swipe) when a user is playing a virtual game.

In some embodiments, the light guide of the totem can provide cues foruser interactions. For example, the halo can be used to inform a user atype and a location of an available user interaction or to indicatecurrent user interactions. As an example, a light emitting diode (LED)underneath the light guide may light up to indicate to the user that theuser can touch or tap the portion of the touch surface where the LEDlights up to select a virtual element on a wearable display. The LED mayalso be used in connection with haptic feedback (e.g., provided by ahaptic actuator in the totem) or audio feedback (e.g., provided by aspeaker of a wearable device) to provide indications of userinteractions or to guide user interactions.

When the totem is used with a wearable device, the user interfaceexperience can be extended to the 3D environment surrounding the user.However, the user's field of view (FOV) perceived through the wearabledisplay may be smaller than the natural FOV of the human eye or smallerthan the entire environment surrounding the user. Thus, there may bephysical or virtual objects in the user's environment that are initiallyoutside the FOV of the augmented reality display but which maysubsequently move into the FOV of the wearable display (e.g., objectsthat can move relative to the user) or may subsequently becomeperceivable if the user's body, head, or eye pose changes (which willchange the user's FOV). For example, in the context of a game, the usermay be trying to find an avatar of a robot. If the robot is just outsidethe current FOV of the user, the user may receive no cues from thewearable display that the robot is nearby. If the user moves his or herhead slightly, the robot may suddenly enter the user's FOV, which may bestartling to the user. Further, if the user's FOV through the wearabledisplay is relatively small, it may be difficult for the user to findthe robot unless the user turns her head or gaze directly at the robot.

To improve the user interface experience, the totem may provideinformation about the objects outside of the user's FOV. For example,the totem can provide, on the outer region of the touchpad, a visualhalo (e.g., emitted via the light guide) for a corresponding objectoutside of the user's current FOV. The light placement or the movementpattern of the halo can be used to indicate information associated withthe object, for example, a brighter or larger halo may indicate theobject is closer to the FOV whereas a dimmer or smaller halo mayindicate the object is farther from the FOV. Similarly, the color of thehalo may be used to indicate the type of the object. For example, acompetitor avatar (in a virtual game) may be associated with a red halowhile a teammate's avatar (in the virtual game) may be associated with agreen halo. As another example, a flashing iridescent halo may indicatea system notification or warning. The light patterns of the halo maychange as the object in the user's environment changes or as the userchanges the pose.

Additionally or alternatively, the light patterns of the halo may beused indicate the progress of a process. For example, while the totem ischarging, the totem may display a halo which corresponds to thepercentage of the charge for the battery. For example, when the batteryis only 25% charged, the totem may display ¼ of a halo (e.g., a 90degree arc). When the battery is charged to 100%, the totem may displaythe entire halo. The light placement or movement patterns of the halomay also provide an indication of the user's interactions to a person inthe user's environment. For example, when the user is recording a videousing an augmented realty device, the LED halo may blink red toreinforce to others nearby that the display is in the recording mode sothat others nearby will not accidentally interrupt or interfere with theuser's recording session.

Embodiments of the totem described herein may be programmable. Forexample, the placement or the movement light patterns may be customizedby an application developer or by a user, in various embodiments. Thehalo may be customized based on the types of applications (e.g., via anapplication programming interface (API)) that the user is interactingwith. As an example, the light guide of the touchpad may be mapped to a4-way d-pad (which corresponds to up, down, left, and right userinterface operations) when a user is using a browser. As anotherexample, the outer region of the touch surface may be mapped to a 3-wayd-pad when the user is pacing a racing game, where the 3-way d-pad maycorrespond to left turn, right turn, and brake. A user may alsocustomize the light placement or movement patterns of the halo. Forexample, the user can turn off the halo or change the color of the haloassociated with receiving email notifications. The user can customizelight patterns associated with the halo using the wearable display or byactuating the totem. Detailed examples of the totem, the halo, and userinterface interactions using the totem and the wearable device aredescribed below.

Although the example totems are described as being used together with awearable system (or any type of AR, MR, or VR device), the exampletotems and techniques described herein can also be used with othersystems. For example, the totems may be used to interact with aprojector or a display (e.g., a television or computer display), agaming system, an audio system, connectable devices in the Internet ofThings (IoT), or another computing device.

Examples of 3D Display of a Wearable System

A wearable system (also referred to herein as an augmented reality (AR)system) can be configured to present 2D or 3D virtual images to a user.The images may be still images, frames of a video, or a video, incombination or the like. The wearable system may comprise a wearabledevice that can present a VR, AR, or MR environment, alone or incombination, for user interaction. The wearable device can include awearable display device, such as, e.g., a head-mounted display (HMD).The wearable device can also include a beltpack which may comprise acentral processing unit to handle some of the data processing for thewearable device, a battery, etc. In some situations, the wearable devicecan be used in interchangeably with an augmented reality device (ARD).

FIG. 1 depicts an illustration of a mixed reality scenario with certainvirtual reality objects, and certain physical objects viewed by aperson. In FIG. 1 , an MR scene 100 is depicted wherein a user of an MRtechnology sees a real-world park-like setting 110 featuring people,trees, buildings in the background, and a concrete platform 120. Inaddition to these items, the user of the MR technology also perceivesthat he or she “sees” a robot statue 130 standing upon the real-worldplatform 120, and a cartoon-like avatar character 140 flying by whichseems to be a personification of a bumble bee, even though theseelements do not exist in the real world.

In order for the 3D display to produce a true sensation of depth, andmore specifically, a simulated sensation of surface depth, it isdesirable for each point in the display's visual field to generate theaccommodative response corresponding to its virtual depth. If theaccommodative response to a display point does not correspond to thevirtual depth of that point, as determined by the binocular depth cuesof convergence and stereopsis, the human eye may experience anaccommodation conflict, resulting in unstable imaging, harmful eyestrain, headaches, and, in the absence of accommodation information,almost a complete lack of surface depth.

VR, AR, and MR experiences can be provided by display systems havingdisplays in which images corresponding to a plurality of depth planesare provided to a viewer. The images may be different for each depthplane (e.g., provide slightly different presentations of a scene orobject) and may be separately focused by the viewer's eyes, therebyhelping to provide the user with depth cues based on the accommodationof the eye required to bring into focus different image features for thescene located on different depth plane and/or based on observingdifferent image features on different depth planes being out of focus.As discussed elsewhere herein, such depth cues provide credibleperceptions of depth.

FIG. 2 illustrates an example of wearable system 200. The wearablesystem 200 can include a display 220, and various mechanical andelectronic modules and systems to support the functioning of display220. The display 220 may be coupled to a frame 230, which is wearable bya user, wearer, or viewer 210. The display 220 can be positioned infront of the eyes of the user 210. A portion of the wearable system(such as the display 220) may be worn on the head of the user.

In FIG. 2 , a speaker 240 is coupled to the frame 230 and positionedadjacent the ear canal of the user (in some embodiments, anotherspeaker, not shown, is positioned adjacent the other ear canal of theuser to provide for stereo/shapeable sound control). The wearable system200 can also include an outward-facing imaging system 464 (shown in FIG.4 ) which observes the world in the environment around the user. Thewearable system 100 can also include an inward-facing imaging system 462(shown in FIG. 4 ) which can track the eye movements of the user. Theinward-facing imaging system may track either one eye's movements orboth eyes' movements. The inward-facing imaging system may be attachedto the frame 230 and may be in electrical communication with theprocessing modules 260 and/or 270, which may process image informationacquired by the inward-facing imaging system to determine, e.g., thepupil diameters and/or orientations of the eyes or eye pose of the user210.

As an example, the wearable system 200 can use the outward-facingimaging system 464 and/or the inward-facing imaging system 462 toacquire images of a pose of the user. The pose may be used to determinea user's motion or synthesize an image of the user. The images acquiredby the outward-facing imaging system 464 and/or the inward-facingimaging system 462 may be communicated to a second user in atelepresence session to create a tangible sense of the user's presencein the second user environment.

The display 220 can be operatively coupled 250, such as by a wired leador wireless connectivity, to a local data processing module 260 whichmay be mounted in a variety of configurations, such as fixedly attachedto the frame 230, fixedly attached to a helmet or hat worn by the user,embedded in headphones, or otherwise removably attached to the user 210(e.g., in a backpack-style configuration, in a belt-coupling styleconfiguration).

The local processing and data module 260 may comprise a hardwareprocessor, as well as digital memory, such as non-volatile memory (e.g.,flash memory), both of which may be utilized to assist in theprocessing, caching, and storage of data. The data may include data a)captured from sensors (which may be, e.g., operatively coupled to theframe 230 or otherwise attached to the user 210), such as image capturedevices (e.g., cameras in the inward-facing imaging system and/or theoutward-facing imaging system), microphones, inertial measurement units(IMUs), accelerometers, compasses, global positioning system (GPS)units, radio devices, and/or gyroscopes; and/or b) acquired and/orprocessed using remote processing module 270 and/or remote datarepository 280, possibly for passage to the display 220 after suchprocessing or retrieval. The local processing and data module 260 may beoperatively coupled by communication links 262 and/or 264, such as viawired or wireless communication links, to the remote processing module270 and/or remote data repository 280 such that these remote modules areavailable as resources to the local processing and data module 260. Inaddition, remote processing module 280 and remote data repository 280may be operatively coupled to each other. The local processing & datamodule 260, the remote processing module 270, and the remote datarepository 280 may each include a network interface to provide thecommunication over the communication links 262, 264.

In some embodiments, the remote processing module 270 may comprise oneor more processors configured to analyze and process data and/or imageinformation. In some embodiments, the remote data repository 280 maycomprise a digital data storage facility, which may be available throughthe internet or other networking configuration in a “cloud” resourceconfiguration. In some embodiments, all data is stored and allcomputations are performed in the local processing and data module,allowing fully autonomous use from a remote module.

The human visual system is complicated and providing a realisticperception of depth is challenging. Without being limited by theory, itis believed that viewers of an object may perceive the object as beingthree-dimensional due to a combination of vergence and accommodation.Vergence movements (i.e., rolling movements of the pupils toward or awayfrom each other to converge the lines of sight of the eyes to fixateupon an object) of the two eyes relative to each other are closelyassociated with focusing (or “accommodation”) of the lenses of the eyes.Under normal conditions, changing the focus of the lenses of the eyes,or accommodating the eyes, to change focus from one object to anotherobject at a different distance will automatically cause a matchingchange in vergence to the same distance, under a relationship known asthe “accommodation-vergence reflex.” Likewise, a change in vergence willtrigger a matching change in accommodation, under normal conditions.Display systems that provide a better match between accommodation andvergence may form more realistic and comfortable simulations ofthree-dimensional imagery.

FIG. 3 illustrates aspects of an approach for simulatingthree-dimensional imagery using multiple depth planes. With reference toFIG. 3 , objects at various distances from eyes 302 and 304 on thez-axis are accommodated by the eyes 302 and 304 so that those objectsare in focus. The eyes 302 and 304 assume particular accommodated statesto bring into focus objects at different distances along the z-axis.Consequently, a particular accommodated state may be said to beassociated with a particular one of depth planes 306, with has anassociated focal distance, such that objects or parts of objects in aparticular depth plane are in focus when the eye is in the accommodatedstate for that depth plane. In some embodiments, three-dimensionalimagery may be simulated by providing different presentations of animage for each of the eyes 302 and 304, and also by providing differentpresentations of the image corresponding to each of the depth planes.While shown as being separate for clarity of illustration, it will beappreciated that the fields of view of the eyes 302 and 304 may overlap,for example, as distance along the z-axis increases. In addition, whileshown as flat for ease of illustration, it will be appreciated that thecontours of a depth plane may be curved in physical space, such that allfeatures in a depth plane are in focus with the eye in a particularaccommodated state. Without being limited by theory, it is believed thatthe human eye typically can interpret a finite number of depth planes toprovide depth perception. Consequently, a highly believable simulationof perceived depth may be achieved by providing, to the eye, differentpresentations of an image corresponding to each of these limited numberof depth planes.

Waveguide Stack Assembly

FIG. 4 illustrates an example of a waveguide stack for outputting imageinformation to a user. A wearable system 400 includes a stack ofwaveguides, or stacked waveguide assembly 480 that may be utilized toprovide three-dimensional perception to the eye/brain using a pluralityof waveguides 432 b, 434 b, 436 b, 438 b, 400 b. In some embodiments,the wearable system 400 may correspond to wearable system 200 of FIG. 2, with FIG. 4 schematically showing some parts of that wearable system200 in greater detail. For example, in some embodiments, the waveguideassembly 480 may be integrated into the display 220 of FIG. 2 .

With continued reference to FIG. 4 , the waveguide assembly 480 may alsoinclude a plurality of features 458, 456, 454, 452 between thewaveguides. In some embodiments, the features 458, 456, 454, 452 may belenses. In other embodiments, the features 458, 456, 454, 452 may not belenses. Rather, they may simply be spacers (e.g., cladding layers and/orstructures for forming air gaps).

The waveguides 432 b, 434 b, 436 b, 438 b, 440 b and/or the plurality oflenses 458, 456, 454, 452 may be configured to send image information tothe eye with various levels of wavefront curvature or light raydivergence. Each waveguide level may be associated with a particulardepth plane and may be configured to output image informationcorresponding to that depth plane. Image injection devices 420, 422,424, 426, 428 may be utilized to inject image information into thewaveguides 440 b, 438 b, 436 b, 434 b, 432 b, each of which may beconfigured to distribute incoming light across each respectivewaveguide, for output toward the eye 410. Light exits an output surfaceof the image injection devices 420, 422, 424, 426, 428 and is injectedinto a corresponding input edge of the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, a single beam of light (e.g., acollimated beam) may be injected into each waveguide to output an entirefield of cloned collimated beams that are directed toward the eye 410 atparticular angles (and amounts of divergence) corresponding to the depthplane associated with a particular waveguide.

In some embodiments, the image injection devices 420, 422, 424, 426, 428are discrete displays that each produce image information for injectioninto a corresponding waveguide 440 b, 438 b, 436 b, 434 b, 432 b,respectively. In some other embodiments, the image injection devices420, 422, 424, 426, 428 are the output ends of a single multiplexeddisplay which may, e.g., pipe image information via one or more opticalconduits (such as fiber optic cables) to each of the image injectiondevices 420, 422, 424, 426, 428.

A controller 460 controls the operation of the stacked waveguideassembly 480 and the image injection devices 420, 422, 424, 426, 428.The controller 460 includes programming (e.g., instructions in anon-transitory computer-readable medium) that regulates the timing andprovision of image information to the waveguides 440 b, 438 b, 436 b,434 b, 432 b. In some embodiments, the controller 460 may be a singleintegral device, or a distributed system connected by wired or wirelesscommunication channels. The controller 460 may be part of the processingmodules 260 and/or 270 (illustrated in FIG. 2 ) in some embodiments.

The waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be configured topropagate light within each respective waveguide by total internalreflection (TIR). The waveguides 440 b, 438 b, 436 b, 434 b, 432 b mayeach be planar or have another shape (e.g., curved), with major top andbottom surfaces and edges extending between those major top and bottomsurfaces. In the illustrated configuration, the waveguides 440 b, 438 b,436 b, 434 b, 432 b may each include light extracting optical elements440 a, 438 a, 436 a, 434 a, 432 a that are configured to extract lightout of a waveguide by redirecting the light, propagating within eachrespective waveguide, out of the waveguide to output image informationto the eye 410. Extracted light may also be referred to as outcoupledlight, and light extracting optical elements may also be referred to asoutcoupling optical elements. An extracted beam of light is outputted bythe waveguide at locations at which the light propagating in thewaveguide strikes a light redirecting element. The light extractingoptical elements (440 a, 438 a, 436 a, 434 a, 432 a) may, for example,be reflective and/or diffractive optical features. While illustrateddisposed at the bottom major surfaces of the waveguides 440 b, 438 b,436 b, 434 b, 432 b for ease of description and drawing clarity, in someembodiments, the light extracting optical elements 440 a, 438 a, 436 a,434 a, 432 a may be disposed at the top and/or bottom major surfaces,and/or may be disposed directly in the volume of the waveguides 440 b,438 b, 436 b, 434 b, 432 b. In some embodiments, the light extractingoptical elements 440 a, 438 a, 436 a, 434 a, 432 a may be formed in alayer of material that is attached to a transparent substrate to formthe waveguides 440 b, 438 b, 436 b, 434 b, 432 b. In some otherembodiments, the waveguides 440 b, 438 b, 436 b, 434 b, 432 b may be amonolithic piece of material and the light extracting optical elements440 a, 438 a, 436 a, 434 a, 432 a may be formed on a surface and/or inthe interior of that piece of material.

With continued reference to FIG. 4 , as discussed herein, each waveguide440 b, 438 b, 436 b, 434 b, 432 b is configured to output light to forman image corresponding to a particular depth plane. For example, thewaveguide 432 b nearest the eye may be configured to deliver collimatedlight, as injected into such waveguide 432 b, to the eye 410. Thecollimated light may be representative of the optical infinity focalplane. The next waveguide up 434 b may be configured to send outcollimated light which passes through the first lens 452 (e.g., anegative lens) before it can reach the eye 410. First lens 452 may beconfigured to create a slight convex wavefront curvature so that theeye/brain interprets light coming from that next waveguide up 434 b ascoming from a first focal plane closer inward toward the eye 410 fromoptical infinity. Similarly, the third up waveguide 436 b passes itsoutput light through both the first lens 452 and second lens 454 beforereaching the eye 410. The combined optical power of the first and secondlenses 452 and 454 may be configured to create another incrementalamount of wavefront curvature so that the eye/brain interprets lightcoming from the third waveguide 436 b as coming from a second focalplane that is even closer inward toward the person from optical infinitythan was light from the next waveguide up 434 b.

The other waveguide layers (e.g., waveguides 438 b, 440 b) and lenses(e.g., lenses 456, 458) are similarly configured, with the highestwaveguide 440 b in the stack sending its output through all of thelenses between it and the eye for an aggregate focal powerrepresentative of the closest focal plane to the person. To compensatefor the stack of lenses 458, 456, 454, 452 when viewing/interpretinglight coming from the world 470 on the other side of the stackedwaveguide assembly 480, a compensating lens layer 430 may be disposed atthe top of the stack to compensate for the aggregate power of the lensstack 458, 456, 454, 452 below. Such a configuration provides as manyperceived focal planes as there are available waveguide/lens pairings.Both the light extracting optical elements of the waveguides and thefocusing aspects of the lenses may be static (e.g., not dynamic orelectro-active). In some alternative embodiments, either or both may bedynamic using electro-active features.

With continued reference to FIG. 4 , the light extracting opticalelements 440 a, 438 a, 436 a, 434 a, 432 a may be configured to bothredirect light out of their respective waveguides and to output thislight with the appropriate amount of divergence or collimation for aparticular depth plane associated with the waveguide. As a result,waveguides having different associated depth planes may have differentconfigurations of light extracting optical elements, which output lightwith a different amount of divergence depending on the associated depthplane. In some embodiments, as discussed herein, the light extractingoptical elements 440 a, 438 a, 436 a, 434 a, 432 a may be volumetric orsurface features, which may be configured to output light at specificangles. For example, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a may be volume holograms, surface holograms, and/ordiffraction gratings. Light extracting optical elements, such asdiffraction gratings, are described in U.S. Patent Publication No.2015/0178939, published Jun. 25, 2015, which is incorporated byreference herein in its entirety.

In some embodiments, the light extracting optical elements 440 a, 438 a,436 a, 434 a, 432 a are diffractive features that form a diffractionpattern, or “diffractive optical element” (also referred to herein as a“DOE”). Preferably, the DOE's have a relatively low diffractionefficiency so that only a portion of the light of the beam is deflectedaway toward the eye 410 with each intersection of the DOE, while therest continues to move through a waveguide via total internalreflection. The light carrying the image information is thus dividedinto a number of related exit beams that exit the waveguide at amultiplicity of locations and the result is a fairly uniform pattern ofexit emission toward the eye 304 for this particular collimated beambouncing around within a waveguide.

In some embodiments, one or more DOEs may be switchable between “on”states in which they actively diffract, and “off” states in which theydo not significantly diffract. For instance, a switchable DOE maycomprise a layer of polymer dispersed liquid crystal, in whichmicrodroplets comprise a diffraction pattern in a host medium, and therefractive index of the microdroplets can be switched to substantiallymatch the refractive index of the host material (in which case thepattern does not appreciably diffract incident light) or themicrodroplet can be switched to an index that does not match that of thehost medium (in which case the pattern actively diffracts incidentlight).

In some embodiments, the number and distribution of depth planes and/ordepth of field may be varied dynamically based on the pupil sizes and/ororientations of the eyes of the viewer. Depth of field may changeinversely with a viewer's pupil size. As a result, as the sizes of thepupils of the viewer's eyes decrease, the depth of field increases suchthat one plane not discernible because the location of that plane isbeyond the depth of focus of the eye may become discernible and appearmore in focus with reduction of pupil size and commensurate increase indepth of field. Likewise, the number of spaced apart depth planes usedto present different images to the viewer may be decreased withdecreased pupil size. For example, a viewer may not be able to clearlyperceive the details of both a first depth plane and a second depthplane at one pupil size without adjusting the accommodation of the eyeaway from one depth plane and to the other depth plane. These two depthplanes may, however, be sufficiently in focus at the same time to theuser at another pupil size without changing accommodation.

In some embodiments, the display system may vary the number ofwaveguides receiving image information based upon determinations ofpupil size and/or orientation, or upon receiving electrical signalsindicative of particular pupil sizes and/or orientations. For example,if the user's eyes are unable to distinguish between two depth planesassociated with two waveguides, then the controller 460 may beconfigured or programmed to cease providing image information to one ofthese waveguides. Advantageously, this may reduce the processing burdenon the system, thereby increasing the responsiveness of the system. Inembodiments in which the DOEs for a waveguide are switchable between onand off states, the DOEs may be switched to the off state when thewaveguide does receive image information.

In some embodiments, it may be desirable to have an exit beam meet thecondition of having a diameter that is less than the diameter of the eyeof a viewer. However, meeting this condition may be challenging in viewof the variability in size of the viewer's pupils. In some embodiments,this condition is met over a wide range of pupil sizes by varying thesize of the exit beam in response to determinations of the size of theviewer's pupil. For example, as the pupil size decreases, the size ofthe exit beam may also decrease. In some embodiments, the exit beam sizemay be varied using a variable aperture.

The wearable system 400 can include an outward-facing imaging system 464(e.g., a digital camera) that images a portion of the world 470. Thisportion of the world 470 may be referred to as the field of view (FOV)and the imaging system 464 is sometimes referred to as an FOV camera.The entire region available for viewing or imaging by a viewer may bereferred to as the field of regard (FOR). The FOR may include 4πsteradians of solid angle surrounding the wearable system 400. In someimplementations of the wearable system 400, the FOR may includesubstantially all of the solid angle around a user of the display system400, because the user can move their head and eyes to look at objectssurrounding the user (in front, in back, above, below, or on the sidesof the user). Images obtained from the outward-facing imaging system 464can be used to track gestures made by the user (e.g., hand or fingergestures), detect objects in the world 470 in front of the user, and soforth.

The wearable system 400 can also include an inward-facing imaging system466 (e.g., a digital camera), which observes the movements of the user,such as the eye movements and the facial movements. The inward-facingimaging system 466 may be used to capture images of the eye 410 todetermine the size and/or orientation of the pupil of the eye 304. Theinward-facing imaging system 466 can be used to obtain images for use indetermining the direction the user is looking (e.g., eye pose) or forbiometric identification of the user (e.g., via iris identification). Insome embodiments, at least one camera may be utilized for each eye, toseparately determine the pupil size and/or eye pose of each eyeindependently, thereby allowing the presentation of image information toeach eye to be dynamically tailored to that eye. In some otherembodiments, the pupil diameter and/or orientation of only a single eye410 (e.g., using only a single camera per pair of eyes) is determinedand assumed to be similar for both eyes of the user. The images obtainedby the inward-facing imaging system 466 may be analyzed to determine theuser's eye pose and/or mood, which can be used by the wearable system400 to decide which audio or visual content should be presented to theuser. The wearable system 400 may also determine head pose (e.g., headposition or head orientation) using sensors such as IMUs (e.g.,accelerometers, gyroscopes, etc.).

The wearable system 400 can include a user input device 466 by which theuser can input commands to the controller 460 to interact with thewearable system 400. For example, the user input device 466 can includea trackpad, a touchscreen, a joystick, a multiple degree-of-freedom(DOF) controller, a capacitive sensing device, a game controller, akeyboard, a mouse, a directional pad (D-pad), a wand, a haptic device, atotem, a smartphone, a smartwatch, a tablet, and so forth, incombination or the like. A multi-DOF controller can sense user input insome or all possible translations (e.g., left/right, forward/backward,or up/down) or rotations (e.g., yaw, pitch, or roll) of the controller.The user can interact with the user input device 466 or objects (e.g.,virtual or physical objects) in his or her environment by, e.g., byclicking on a mouse, tapping on a touch pad, swiping on a touch screen,hovering over or touching a capacitive button, pressing a key on akeyboard or a game controller (e.g., a 5-way d-pad), pointing ajoystick, wand, or totem toward the object, pressing a button on aremote control, or other interactions with a user input device. Theactuation of the user input device 466 may cause the wearable system toperform a user interface operation, such as, e.g., displaying a virtualuser interface menu associated with an object, animating the user'savatar in a game, etc. As described herein, the user input device 466may be configured to emit light. The light patterns may representinformation associated with an object in the user's environment, theuser's interaction with the user input device 466 or a wearable device,and so on.

In some cases, the user may use a finger (e.g., a thumb) to press orswipe on a touch-sensitive input device to provide input to the wearablesystem 400 (e.g., to provide user input to a user interface provided bythe wearable system 400). The user input device 466 may be held by theuser's hand during the use of the wearable system 400. The user inputdevice 466 can be in wired or wireless communication with the wearablesystem 400. The user input device 466 may comprise embodiments of thetotem described herein. The totem can include a touch surface which canallow a user to actuate the totem by swiping along a trajectory ortapping, etc.

FIG. 5 shows an example of exit beams outputted by a waveguide. Onewaveguide is illustrated, but it will be appreciated that otherwaveguides in the waveguide assembly 480 may function similarly, wherethe waveguide assembly 480 includes multiple waveguides. Light 520 isinjected into the waveguide 432 b at the input edge 432 c of thewaveguide 432 b and propagates within the waveguide 432 b by TIR. Atpoints where the light 520 impinges on the DOE 432 a, a portion of thelight exits the waveguide as exit beams 510. The exit beams 510 areillustrated as substantially parallel but they may also be redirected topropagate to the eye 410 at an angle (e.g., forming divergent exitbeams), depending on the depth plane associated with the waveguide 432b. It will be appreciated that substantially parallel exit beams may beindicative of a waveguide with light extracting optical elements thatoutcouple light to form images that appear to be set on a depth plane ata large distance (e.g., optical infinity) from the eye 410. Otherwaveguides or other sets of light extracting optical elements may outputan exit beam pattern that is more divergent, which would require the eye410 to accommodate to a closer distance to bring it into focus on theretina and would be interpreted by the brain as light from a distancecloser to the eye 410 than optical infinity.

FIG. 6 is a schematic diagram showing an optical system including awaveguide apparatus, an optical coupler subsystem to optically couplelight to or from the waveguide apparatus, and a control subsystem, usedin the generation of a multi-focal volumetric display, image, or lightfield. The optical system can include a waveguide apparatus, an opticalcoupler subsystem to optically couple light to or from the waveguideapparatus, and a control subsystem. The optical system can be used togenerate a multi-focal volumetric, image, or light field. The opticalsystem can include one or more primary planar waveguides 632 a (only oneis shown in FIG. 6 ) and one or more DOEs 632 b associated with each ofat least some of the primary waveguides 632 a. The planar waveguides 632b can be similar to the waveguides 432 b, 434 b, 436 b, 438 b, 440 bdiscussed with reference to FIG. 4 . The optical system may employ adistribution waveguide apparatus to relay light along a first axis(vertical or Y-axis in view of FIG. 6 ), and expand the light'seffective exit pupil along the first axis (e.g., Y-axis). Thedistribution waveguide apparatus, may, for example include adistribution planar waveguide 622 b and at least one DOE 622 a(illustrated by double dash-dot line) associated with the distributionplanar waveguide 622 b. The distribution planar waveguide 622 b may besimilar or identical in at least some respects to the primary planarwaveguide 632 b, having a different orientation therefrom. Likewise, atleast one DOE 622 a may be similar or identical in at least somerespects to the DOE 632 a. For example, the distribution planarwaveguide 622 b and/or DOE 622 a may be comprised of the same materialsas the primary planar waveguide 632 b and/or DOE 632 a, respectively.Embodiments of the optical display system 600 shown in FIG. 6 can beintegrated into the wearable system 200 shown in FIG. 2 .

The relayed and exit-pupil expanded light is optically coupled from thedistribution waveguide apparatus into the one or more primary planarwaveguides 632 b. The primary planar waveguide 632 b relays light alonga second axis, preferably orthogonal to first axis, (e.g., horizontal orX-axis in view of FIG. 6 ). Notably, the second axis can be anon-orthogonal axis to the first axis. The primary planar waveguide 632b expands the light's effective exit pupil along that second axis (e.g.,X-axis). For example, the distribution planar waveguide 622 b can relayand expand light along the vertical or Y-axis, and pass that light tothe primary planar waveguide 632 b which relays and expands light alongthe horizontal or X-axis.

The optical system may include one or more sources of colored light(e.g., red, green, and blue laser light) 610 which may be opticallycoupled into a proximal end of a single mode optical fiber 640. A distalend of the optical fiber 640 may be threaded or received through ahollow tube 8 of piezoelectric material. The distal end protrudes fromthe tube 642 as fixed-free flexible cantilever 644. The piezoelectrictube 642 can be associated with four quadrant electrodes (notillustrated). The electrodes may, for example, be plated on the outside,outer surface or outer periphery or diameter of the tube 642. A coreelectrode (not illustrated) is also located in a core, center, innerperiphery or inner diameter of the tube 642.

Drive electronics 650, for example electrically coupled via wires 660,drive opposing pairs of electrodes to bend the piezoelectric tube 642 intwo axes independently. The protruding distal tip of the optical fiber644 has mechanical modes of resonance. The frequencies of resonance candepend upon a diameter, length, and material properties of the opticalfiber 644. By vibrating the piezoelectric tube 8 near a first mode ofmechanical resonance of the fiber cantilever 644, the fiber cantilever644 is caused to vibrate, and can sweep through large deflections.

By stimulating resonant vibration in two axes, the tip of the fibercantilever 644 is scanned biaxially in an area filling two dimensional(2-D) scan. By modulating an intensity of light source(s) 610 insynchrony with the scan of the fiber cantilever 644, light emerging fromthe fiber cantilever 644 forms an image. Descriptions of such a set upare provided in U.S. Patent Publication No. 2014/0003762, which isincorporated by reference herein in its entirety.

A component of an optical coupler subsystem collimates the lightemerging from the scanning fiber cantilever 644. The collimated light isreflected by mirrored surface 648 into the narrow distribution planarwaveguide 622 b which contains the at least one diffractive opticalelement (DOE) 622 a. The collimated light propagates vertically(relative to the view of FIG. 6 ) along the distribution planarwaveguide 622 b by total internal reflection (TIR), and in doing sorepeatedly intersects with the DOE 622 a. The DOE 622 a preferably has alow diffraction efficiency. This causes a fraction (e.g., 10%) of thelight to be diffracted toward an edge of the larger primary planarwaveguide 632 b at each point of intersection with the DOE 622 a, and afraction of the light to continue on its original trajectory down thelength of the distribution planar waveguide 622 b via TIR.

At each point of intersection with the DOE 622 a, additional light isdiffracted toward the entrance of the primary waveguide 632 b. Bydividing the incoming light into multiple outcoupled sets, the exitpupil of the light is expanded vertically by the DOE 4 in thedistribution planar waveguide 622 b. This vertically expanded lightcoupled out of distribution planar waveguide 622 b enters the edge ofthe primary planar waveguide 632 b.

Light entering primary waveguide 632 b propagates horizontally (relativeto the view of FIG. 6 ) along the primary waveguide 632 b via TIR. Asthe light intersects with DOE 632 a at multiple points as it propagateshorizontally along at least a portion of the length of the primarywaveguide 632 b via TIR. The DOE 632 a may advantageously be designed orconfigured to have a phase profile that is a summation of a lineardiffraction pattern and a radially symmetric diffractive pattern, toproduce both deflection and focusing of the light. The DOE 632 a mayadvantageously have a low diffraction efficiency (e.g., 10%), so thatonly a portion of the light of the beam is deflected toward the eye ofthe view with each intersection of the DOE 632 a while the rest of thelight continues to propagate through the primary waveguide 632 b viaTIR.

At each point of intersection between the propagating light and the DOE632 a, a fraction of the light is diffracted toward the adjacent face ofthe primary waveguide 632 b allowing the light to escape the TIR, andemerge from the face of the primary waveguide 632 b. In someembodiments, the radially symmetric diffraction pattern of the DOE 632 aadditionally imparts a focus level to the diffracted light, both shapingthe light wavefront (e.g., imparting a curvature) of the individual beamas well as steering the beam at an angle that matches the designed focuslevel.

Accordingly, these different pathways can cause the light to be coupledout of the primary planar waveguide 632 b by a multiplicity of DOEs 632a at different angles, focus levels, and/or yielding different fillpatterns at the exit pupil. Different fill patterns at the exit pupilcan be beneficially used to create a light field display with multipledepth planes. Each layer in the waveguide assembly or a set of layers(e.g., 3 layers) in the stack may be employed to generate a respectivecolor (e.g., red, blue, green). Thus, for example, a first set of threeadjacent layers may be employed to respectively produce red, blue andgreen light at a first focal depth. A second set of three adjacentlayers may be employed to respectively produce red, blue and green lightat a second focal depth. Multiple sets may be employed to generate afull 3D or 4D color image light field with various focal depths.

Other Components of the Wearable System

In many implementations, the wearable system may include othercomponents in addition or in alternative to the components of thewearable system described above. The wearable system may, for example,include one or more haptic devices or components. The haptic device(s)or component(s) may be operable to provide a tactile sensation to auser. For example, the haptic device(s) or component(s) may provide atactile sensation of pressure and/or texture when touching virtualcontent (e.g., virtual objects, virtual tools, other virtualconstructs). The tactile sensation may replicate a feel of a physicalobject which a virtual object represents, or may replicate a feel of animagined object or character (e.g., a dragon) which the virtual contentrepresents. In some implementations, haptic devices or components may beworn by the user (e.g., a user wearable glove). In some implementations,haptic devices or components may be held by the user.

The wearable system may, for example, include one or more physicalobjects which are manipulable by the user to allow input or interactionwith the wearable system. These physical objects may be referred toherein as totems. Some totems may take the form of inanimate objects,such as for example, a piece of metal or plastic, a wall, a surface oftable. In certain implementations, the totems may not actually have anyphysical input structures (e.g., keys, triggers, joystick, trackball,rocker switch). Instead, the totem may simply provide a physicalsurface, and the wearable system may render a user interface so as toappear to a user to be on one or more surfaces of the totem. Forexample, the wearable system may render an image of a computer keyboardand trackpad to appear to reside on one or more surfaces of a totem. Forinstance, the wearable system may render a virtual computer keyboard andvirtual trackpad to appear on a surface of a thin rectangular plate ofaluminum which serves as a totem. The rectangular plate does not itselfhave any physical keys or trackpad or sensors. However, the wearablesystem may detect user manipulation or interaction or touches with therectangular plate as selections or inputs made via the virtual keyboardand/or virtual trackpad. The user input device 466 (shown in FIG. 4 )may be an embodiment of a totem may, which may include a trackpad, atouchpad, a trigger, a joystick, a trackball, a rocker switch, a mouse,a keyboard, a multi-degree-of-freedom controller, or another physicalinput device. A user may use the totem, alone or in combination withposes, to interact with the wearable system and/or other users.

Examples of haptic devices and totems usable with the wearable devices,HMD, ARD, and display systems of the present disclosure are described inU.S. Patent Publication No. 2015/0016777, which is incorporated byreference herein in its entirety.

Example Wearable Systems, Environments, and Interfaces

A wearable system may employ various mapping related techniques in orderto achieve high depth of field in the rendered light fields. In mappingout the virtual world, it is advantageous to know all the features andpoints in the real world to accurately portray virtual objects inrelation to the real world. To this end, FOV images captured from usersof the wearable system can be added to a world model by including newpictures that convey information about various points and features ofthe real world. For example, the wearable system can collect a set ofmap points (such as 2D points or 3D points) and find new map points torender a more accurate version of the world model. The world model of afirst user can be communicated (e.g., over a network such as a cloudnetwork) to a second user so that the second user can experience theworld surrounding the first user.

FIG. 7 is a block diagram of an example of an MR environment 700. The MRenvironment 700 may be configured to receive input (e.g., visual input702 from the user's wearable system, stationary input 704 such as roomcameras, sensory input 706 from various sensors, gestures, totems, eyetracking, user input from the user input device 466, etc.) from varioususer systems 720 a, 720 b. The user systems 720 a, 720 b can compriseone or more user wearable systems (e.g., wearable system 200 and/ordisplay system 220) and/or stationary room systems (e.g., room cameras,etc.). The wearable systems can use various sensors (e.g.,accelerometers, gyroscopes, temperature sensors, movement sensors, depthsensors, GPS sensors, inward-facing imaging system, outward-facingimaging system, etc.) to determine the location and various otherattributes of the environment of the user. This information may furtherbe supplemented with information from stationary cameras in the roomthat may provide images and/or various cues from a different point ofview. The image data acquired by the cameras (such as the room camerasand/or the cameras of the outward-facing imaging system) may be reducedto a set of mapping points.

One or more object recognizers 708 can crawl through the received data(e.g., the collection of points) and recognize and/or map points, tagimages, attach semantic information to objects with the help of a mapdatabase 710. The map database 710 may comprise various points collectedover time and their corresponding objects. The various devices and themap database can be connected to each other through a network (e.g.,LAN, WAN, etc.) to access the cloud.

Based on this information and collection of points in the map database,the object recognizers 708 a to 708 n may recognize objects in anenvironment. For example, the object recognizers can recognize faces,persons, windows, walls, user input devices, televisions, other objectsin the user's environment, etc. One or more object recognizers may bespecialized for object with certain characteristics. For example, theobject recognizer 708 a may be used to recognizer faces, while anotherobject recognizer may be used recognize totems.

The object recognitions may be performed using a variety of computervision techniques. For example, the wearable system can analyze theimages acquired by the outward-facing imaging system 464 (shown in FIG.4 ) to perform scene reconstruction, event detection, video tracking,object recognition, object pose estimation, learning, indexing, motionestimation, or image restoration, etc. One or more computer visionalgorithms may be used to perform these tasks. Non-limiting examples ofcomputer vision algorithms include: Scale-invariant feature transform(SIFT), speeded up robust features (SURF), oriented FAST and rotatedBRIEF (ORB), binary robust invariant scalable keypoints (BRISK), fastretina keypoint (FREAK), Viola-Jones algorithm, Eigenfaces approach,Lucas-Kanade algorithm, Horn-Schunk algorithm, Mean-shift algorithm,visual simultaneous location and mapping (vSLAM) techniques, asequential Bayesian estimator (e.g., Kalman filter, extended Kalmanfilter, etc.), bundle adjustment, Adaptive thresholding (and otherthresholding techniques), Iterative Closest Point (ICP), Semi GlobalMatching (SGM), Semi Global Block Matching (SGBM), Feature PointHistograms, various machine learning algorithms (such as e.g., supportvector machine, k-nearest neighbors algorithm, Naive Bayes, neuralnetwork (including convolutional or deep neural networks), or othersupervised/unsupervised models, etc.), and so forth.

The object recognitions can additionally or alternatively be performedby a variety of machine learning algorithms. Once trained, the machinelearning algorithm can be stored by the HMD. Some examples of machinelearning algorithms can include supervised or non-supervised machinelearning algorithms, including regression algorithms (such as, forexample, Ordinary Least Squares Regression), instance-based algorithms(such as, for example, Learning Vector Quantization), decision treealgorithms (such as, for example, classification and regression trees),Bayesian algorithms (such as, for example, Naive Bayes), clusteringalgorithms (such as, for example, k-means clustering), association rulelearning algorithms (such as, for example, a-priori algorithms),artificial neural network algorithms (such as, for example, Perceptron),deep learning algorithms (such as, for example, Deep Boltzmann Machine,or deep neural network), dimensionality reduction algorithms (such as,for example, Principal Component Analysis), ensemble algorithms (suchas, for example, Stacked Generalization), and/or other machine learningalgorithms. In some embodiments, individual models can be customized forindividual data sets. For example, the wearable device can generate orstore a base model. The base model may be used as a starting point togenerate additional models specific to a data type (e.g., a particularuser in the telepresence session), a data set (e.g., a set of additionalimages obtained of the user in the telepresence session), conditionalsituations, or other variations. In some embodiments, the wearable HMDcan be configured to utilize a plurality of techniques to generatemodels for analysis of the aggregated data. Other techniques may includeusing pre-defined thresholds or data values.

The wearable system can also supplement recognized objects with semanticinformation to give life to the objects. For example, if the objectrecognizer recognizes a set of points to be a door, the system mayattach some semantic information (e.g., the door has a hinge and has a90 degree movement about the hinge). If the object recognizer recognizesa set of points to be a totem, the wearable system may attach semanticinformation that the totem can be paired (e.g., via Bluetooth) with thewearable system. Over time the map database grows as the system (whichmay reside locally or may be accessible through a wireless network)accumulates more data from the world. Once the objects are recognized,the information may be transmitted to one or more wearable systems. Forexample, the MR environment 700 may include information about a scenehappening in California. The environment 700 may be transmitted to oneor more users in New York. Based on data received from an FOV camera andother inputs, the object recognizers and other software components canmap the points collected from the various images, recognize objectsetc., such that the scene may be accurately “passed over” to a seconduser, who may be in a different part of the world. The environment 700may also use a topological map for localization purposes.

FIG. 8 is a process flow diagram of an example of a method 800 ofrendering virtual content in relation to recognized objects. The method800 describes how a virtual scene may be represented to a user of the MRsystem (e.g., a wearable system). The user may be geographically remotefrom the scene. For example, the user may be New York, but may want toview a scene that is presently going on in California, or may want to goon a walk with a friend who resides in California.

At block 810, the wearable system may receive input from the user andother users regarding the environment of the user. This may be achievedthrough various input devices, and knowledge already possessed in themap database. The user's FOV camera, sensors, GPS, eye tracking, etc.,convey information to the system at block 810. The system may determinesparse points based on this information at block 820. The sparse pointsmay be used in determining pose data (e.g., head pose, eye pose, bodypose, and/or hand gestures) that can be used in displaying andunderstanding the orientation and position of various objects in theuser's surroundings. The object recognizers 708 a, 708 n may crawlthrough these collected points and recognize one or more objects using amap database at block 830. This information may then be conveyed to theuser's individual wearable system at block 840, and the desired virtualscene may be accordingly displayed to the user at block 850. Forexample, the desired virtual scene (e.g., user in CA) may be displayedat the appropriate orientation, position, etc., in relation to thevarious objects and other surroundings of the user in New York.

FIG. 9 is a block diagram of another example of a wearable system. Inthis example, the wearable system 900 comprises a map, which may includemap data for the world. The map may partly reside locally on thewearable system, and may partly reside at networked storage locationsaccessible by wired or wireless network (e.g., in a cloud system). Apose process 910 may be executed on the wearable computing architecture(e.g., processing module 260 or controller 460) and utilize data fromthe map to determine position and orientation of the wearable computinghardware or user. Pose data may be computed from data collected on thefly as the user is experiencing the system and operating in the world.The data may comprise images, data from sensors (such as inertialmeasurement devices, which generally comprise accelerometer andgyroscope components) and surface information pertinent to objects inthe real or virtual environment.

A sparse point representation may be the output of a simultaneouslocalization and mapping (SLAM or V-SLAM, referring to a configurationwherein the input is images/visual only) process. The system can beconfigured to not only find out where in the world the variouscomponents are, but what the world is made of. Pose may be a buildingblock that achieves many goals, including populating the map and usingthe data from the map.

In one embodiment, a sparse point position may not be completelyadequate on its own, and further information may be needed to produce amultifocal AR, VR, or MR experience. Dense representations, generallyreferring to depth map information, may be utilized to fill this gap atleast in part. Such information may be computed from a process referredto as Stereo 940, wherein depth information is determined using atechnique such as triangulation or time-of-flight sensing. Imageinformation and active patterns (such as infrared patterns created usingactive projectors) may serve as input to the Stereo process 940. Asignificant amount of depth map information may be fused together, andsome of this may be summarized with a surface representation. Forexample, mathematically definable surfaces are efficient (e.g., relativeto a large point cloud) and digestible inputs to other processingdevices like game engines. Thus, the output of the Stereo process (e.g.,a depth map) 940 may be combined in the Fusion process 930. Pose may bean input to this Fusion process 930 as well, and the output of Fusion930 becomes an input to populating the map process 920. Sub-surfaces mayconnect with each other, such as in topographical mapping, to formlarger surfaces, and the map becomes a large hybrid of points andsurfaces.

To resolve various aspects in a mixed reality process 960, variousinputs may be utilized. For example, in the embodiment depicted in FIG.9 , Game parameters may be inputs to determine that the user of thesystem is playing a monster battling game with one or more monsters atvarious locations, monsters dying or running away under variousconditions (such as if the user shoots the monster), walls or otherobjects at various locations, and the like. The world map may includeinformation regarding where such objects are relative to each other, tobe another valuable input to mixed reality. Pose relative to the worldbecomes an input as well and plays a key role to almost any interactivesystem.

Controls or inputs from the user are another input to the wearablesystem 900. As described herein, user inputs can include visual input,gestures, totems, audio input, sensory input, etc. In order to movearound or play a game, for example, the user may need to instruct thewearable system 900 regarding what he or she wants to do. Beyond justmoving oneself in space, there are various forms of user controls thatmay be utilized. In one embodiment, a totem, another user input device,or an object such as a toy gun may be held by the user and tracked bythe system. The system preferably will be configured to know that theuser is holding the item and understand what kind of interaction theuser is having with the item (e.g., if the totem or object is a gun, thesystem may be configured to understand location and orientation, as wellas whether the user is clicking a trigger or other sensed button orelement which may be equipped with a sensor, such as an IMU, which mayassist in determining what is going on, even when such activity is notwithin the field of view of any of the cameras.)

Hand gesture tracking or recognition may also provide input information.The wearable system 900 may be configured to track and interpret handgestures for button presses, for gesturing left or right, stop, grab,hold, etc. For example, in one configuration, the user may want to flipthrough emails or a calendar in a non-gaming environment, or do a “fistbump” with another person or player. The wearable system 900 may beconfigured to leverage a minimum amount of hand gesture, which may ormay not be dynamic. For example, the gestures may be simple staticgestures like open hand for stop, thumbs up for ok, thumbs down for notok; or a hand flip right, or left, or up/down for directional commands.

Eye tracking is another input (e.g., tracking where the user is lookingto control the display technology to render at a specific depth orrange). In one embodiment, vergence of the eyes may be determined usingtriangulation, and then using a vergence/accommodation model developedfor that particular person, accommodation may be determined.

With regard to the camera systems, the example wearable system 900 shownin FIG. 9 can include three pairs of cameras: a relative wide FOV orpassive SLAM pair of cameras arranged to the sides of the user's face, adifferent pair of cameras oriented in front of the user to handle theStereo imaging process 940 and also to capture hand gestures andtotem/object tracking in front of the user's face. The cameras in thethree pairs of cameras may be a part of the outward-facing imagingsystem 464 (shown in FIG. 4 ). The wearable system 900 can include eyetracking cameras (which may be a part of an inward-facing imaging system462 shown in FIG. 4 ) oriented toward the eyes of the user in order totriangulate eye vectors and other information. The wearable system 900may also comprise one or more textured light projectors (such asinfrared (IR) projectors) to inject texture into a scene.

FIG. 10 is a process flow diagram of an example of a method 1000 fordetermining user input to a wearable system. In this example, the usermay interact with a totem. The user may have multiple totems. Forexample, the user may have designated one totem for a social mediaapplication, another totem for playing games, etc. At block 1010, thewearable system may detect a motion of a totem. The movement of thetotem may be recognized through the user's FOV camera or may be detectedthrough sensors (e.g., haptic glove, image sensors, hand trackingdevices, eye-tracking cameras, head pose sensors, etc.).

Based at least partly on the detected gesture, eye pose, head pose, orinput through the totem, the wearable system detects a position,orientation, and/or movement of the totem (or the user's eyes or head orgestures) with respect to a reference frame, at block 1020. Thereference frame may be a set of map points based on which the wearablesystem translates the movement of the totem (or the user) to an actionor command. At block 1030, the user's interaction with the totem ismapped. Based on the mapping of the user interaction with respect to thereference frame 1020, the system determines the user input at block1040.

For example, the user may move a totem or physical object back and forthto signify turning a virtual page and moving on to a next page or movingfrom one user interface (UI) display screen to another UI screen. Asanother example, the user may move their head or eyes to look atdifferent real or virtual objects in the user's FOR. If the user's gazeat a particular real or virtual object is longer than a threshold time,the real or virtual object may be selected as the user input. In someimplementations, the vergence of the user's eyes can be tracked and anaccommodation/vergence model can be used to determine the accommodationstate of the user's eyes, which provides information on a depth plane onwhich the user is focusing. In some implementations, the wearable systemcan use raycasting techniques to determine which real or virtual objectsare along the direction of the user's head pose or eye pose. In variousimplementations, the ray casting techniques can include casting thin,pencil rays with substantially little transverse width or casting rayswith substantial transverse width (e.g., cones or frustums).

The user interface may be projected by the display system as describedherein (such as the display 220 in FIG. 2 ). It may also be displayedusing a variety of other techniques such as one or more projectors. Theprojectors may project images onto a physical object such as a canvas ora globe. Interactions with user interface may be tracked using one ormore cameras external to the system or part of the system (such as,e.g., using the inward-facing imaging system 462 or the outward-facingimaging system 464).

FIG. 11 is a process flow diagram of an example of a method 1100 forinteracting with a virtual user interface. The method 1100 may beperformed by the wearable system described herein.

At block 1110, the wearable system may identify a particular UI. Thetype of UI may be predetermined by the user. The wearable system mayidentify that a particular UI needs to be populated based on a userinput (e.g., gesture, visual data, audio data, sensory data, directcommand, etc.). At block 1120, the wearable system may generate data forthe virtual UI. For example, data associated with the confines, generalstructure, shape of the UI etc., may be generated. In addition, thewearable system may determine map coordinates of the user's physicallocation so that the wearable system can display the UI in relation tothe user's physical location. For example, if the UI is body centric,the wearable system may determine the coordinates of the user's physicalstance, head pose, or eye pose such that a ring UI can be displayedaround the user or a planar UI can be displayed on a wall or in front ofthe user. If the UI is hand centric, the map coordinates of the user'shands may be determined. These map points may be derived through datareceived through the FOV cameras, sensory input, or any other type ofcollected data.

At block 1130, the wearable system may send the data to the display fromthe cloud or the data may be sent from a local database to the displaycomponents. At block 1140, the UI is displayed to the user based on thesent data. For example, a light field display can project the virtual UIinto one or both of the user's eyes. Once the virtual UI has beencreated, the wearable system may simply wait for a command from the userto generate more virtual content on the virtual UI at block 1150. Forexample, the UI may be a body centric ring around the user's body. Thewearable system may then wait for the command (a gesture, a head or eyemovement, input from a user input device, etc.), and if it is recognized(block 1160), virtual content associated with the command may bedisplayed to the user (block 1170).

Additional examples of wearable systems, UI, and user experiences (UX)are described in U.S. Patent Publication No. 2015/0016777, which isincorporated by reference herein in its entirety.

Overview of an Example Totem

As described with reference to FIGS. 4 and 7-10 , a user can performvarious user interface operations on a display (e.g., display 220) usingthe user input device 466 (such as, e.g., a totem). FIG. 12A illustratesan example embodiment of a totem 1200, showing side 1210 and front(user-facing) 1250 views. The totem 1200 may be the user input device466 (shown in FIG. 4 ) alone or in combination with other user inputdevices. The totem 1200 can be sized and shaped so as to be handheld.Further examples of user input devices (such as, e.g., totems) in one ormore of the various embodiments disclosed herein are shown and describedin U.S. patent application Ser. No. 15/683,677, filed Aug. 22, 2017,with ornamental appearances the same as or similar to the totemcontroller of U.S. Design patent application Ser. No. 29/575,031, filedAug. 22, 2016, both of the aforementioned applications are herebyincorporated by reference herein in their entireties.

The totem 1200 shown in FIG. 12A can include a body 1214 having atrigger 1212, a home button 1256, and a touchpad 1260, although more orfewer buttons, triggers, or features can be included in other exampletotems. A light guide 1264 can substantially surround the touchpad 1260.In the examples shown in FIGS. 12A and 12B, the light guide 1264 issubstantially annular (e.g., a circular ring) that substantiallysurrounds a circular touchpad 1260. In other embodiments, the touchpad1260 and the light guide 1264 can be shaped differently. For example,the touchpad 1260 can be polygonally shaped (e.g., square, rectangular,hexagonal, etc.) or oval, and the light guide can have a shape (e.g.,circular, oval, polygonal, etc.) that substantially surrounds thetouchpad.

The trigger, home button, and touchpad can accept user inputs (e.g., bybeing pulled, depressed, or touched, respectively). The light guide 1264can be illuminated to display light patterns. In some embodiments, thelight guide 1264 is touch sensitive and can receive user input. Thetotem 1200 can be removably attached to a base 1220 when not being heldby the user. The base 1220 may include an electrical power connection(e.g., to a wall socket) and can be used to charge the totem 1200 whenthe totem is attached to the base 1220.

Examples of a Trigger

The trigger 1212 can be located on the upper portion of the totem body1214 that faces away from the user. The trigger 1212 can include a touchsurface 1234 and a touch sensor (not shown in FIG. 12A), which canreceive a user input. A touch sensor can sense a user's finger on (ornear) the touch surface 1234 and the movement of the user's finger onthe touch surface 1234. Additionally or alternatively, the trigger 1212can include a button 1232 which the user can press. The button 1232 caninclude a pressure sensor which can detect when a user presses thebutton 1232. The button 1232 may be pressed 6-8 mm from its restingposition. In some embodiments, the trigger 1212 can include multiplebuttons such as, e.g., select, back, option, etc.

The trigger 1212 may be implemented, for example, using an Alps forcesensor. The trigger 1212 (or the button 1232) can also be implementedusing an analog or digital button. The trigger 1212 may be associatedwith a microprocessor. The microprocessor may reserve twogeneral-purpose input/output (GPIO) pins for various functions of thetrigger 1212.

In some implementations, the trigger 1212 can be configured to providehaptic feedback. The trigger 1212 can include a haptic actuator, suchas, e.g., linear resonant actuator (LRA), eccentric rotating mass (ERM),piezo actuator, etc. For example, the trigger 1212 can employ an S-TypeLRA (such as, e.g., an S-type Alps haptic on an analog button 1232) togenerate a vibration.

A user can actuate the trigger 1212 using various hand gestures. Forexample, the user can actuate the trigger 1212 by touching, swiping,tapping, pressing, etc. The trigger 1212 can provide rich touch featureswhere various user interface interactions may be associated withdifferent hand gestures used to actuate the trigger 1212. For example, auser can switch between an AR interface and a VR interface by pressingthe trigger 1212. As another example, the user can swipe the virtualobjects in and out of the user's FOV by swiping on the touch surface1234 of the trigger. The user interface interactions can also beassociated with the duration of the actuation. The totem 1200 can recordthe duration of the press and determine a user interface operation basedon the duration of the press. For example, pressing the trigger 1212 foran extended duration (such as e.g., for 3-5 seconds) may cause thewearable system to exit a program (such as e.g., a movie) while a quickpress of the trigger may cause the wearable system to select a virtualobject in the user's direction of gaze.

The trigger may be actuated in conjunction with other components of thetotem or a pose of the user to perform a user interface operation. Forexample, the user may press the trigger 1212 while swiping on thetouchpad 1260 to move a virtual object. As another example, the user canpress the trigger 1212 with his or her right thumb and move his or herarm to the right to move a virtual object rightward.

The trigger 1212 can also be configured to provide an indication to theuser that one or more user interface operations are available. Forexample, when the trigger 1212 is pressed for an extended duration, thewearable system may enable a user to change the settings of a game. Toindicate to the user that he or she can now change the settings, thewearable system may provide a haptic feedback (such as, e.g., avibration) on the trigger 1212 (or on the body of the totem). Inaddition to or in alternative to providing an indication that an userinterface operation are available, the haptic feedback can also be usedto inform the user that he or she has actuated the trigger 1212 using acertain gesture (such as e.g., tapping or pressing).

In other embodiments, some or all of the above-described functionalityof the trigger 1212 can be implemented by other buttons ortouch-sensitive surfaces of the totem 1200. For example, as describedbelow with reference to FIG. 17B, a button (also referred to as abumper) can perform some or all of the functionality of the trigger 1212(or the home button 1256).

Other examples of a totem 1200 are described with reference to FIGS. 17Aand 17B.

Examples of a Touchpad

The totem can include a touchpad 1260 located on the upper front portionof the totem body 1214 that faces toward the user. The touchpad 1260 caninclude a touch surface 1262 and a light guide 1264. In some examples,the architecture and/or functionality of the touchpad 1260 may besubstantially the same as or similar to that which is described in U.S.patent application Ser. No. 15/683,677, which as mentioned above isincorporated by reference herein in its entirety. In the illustratedexample, the light guide 1264 substantially surrounds the touch surface1262. The touchpad 1260 can include various interactable regions. Forexample, the user can actuate the touch surface 1262, the light guide1262, alone or in combination. The touch surface 1262 can furtherinclude multiple interactable regions, with each region being mapped toa type of user input (or a user interface operation).

The touch surface 1262 may be a circular surface with a diameter in therange of 27 mm-40 mm. The touch surface 1262 may also be other shapes,such as, e.g., oval, rectangle, triangle, diamond, irregular, etc. Incertain implementations, the touchpad 1260 may have a less than 50 mstraffic-to-pilot (T2P) latency.

The touch surface 1262 can be substantially surrounded by a light guide1264. For example, in various embodiments, the light guide 1264 maysubtend an angular distance around the touch surface of greater than 90degrees, greater than 180 degrees, greater than 270 degrees, or up to afull 360 degrees. The light guide 1264 can be illuminated to display ahalo with various placement and movement of light patterns. The lightguide 1264 can diffuse the light generated by light sources 1330 (e.g.,LEDs) of the touchpad to display the halo, so that the illumination fromindividual, discrete light sources is merged. The light guide 1264 cancomprise a diffusive optical element formed from, e.g., plastic orpolymeric material formed into a ring shape, which can transmit (anddiffuse) the light from the light sources 1330 to a viewer of the totem1200 (see, e.g., FIGS. 13A-14A). The light guide can be transparent ortranslucent. In various embodiments, the light guide 1264 can comprisediffuser sheets, diffuser films, an etched waveguide, a transmissiveoptical element comprising layers of particles, irregular surfaces,holographs, white surfaces, ground glass, polytetrafluoroethylene (PTFEor Teflon), opal glass, greyed glass, colored gel, and so forth.Embodiments of a light guide comprising optically diffusive materialadvantageously may diffuse and spread out the light from the lightsources 1330 so that the light guide appears to have an overall,substantially continuous glow (when all light sources are illuminated)rather than appearing as discrete, individual light sources (if thelight guide were substantially transparent).

The touchpad can include a number of red, green, blue (RGB) LEDs (suchas, e.g., 6-24 RGB LEDs). The light placement or movement patterns ofthe halo can provide visual indications of the user's interactions withthe totem or other components of the wearable system, the status of aprogress associated with the wearable system, the objects in the user'senvironment, etc. See, for example, the LEDs 1383 a of FIG. 13C or theLEDs 1394 a of FIG. 13E.

The light sources can include (additionally or alternatively) emittersthat radiate in non-visible portions of the electromagnetic spectrum,e.g., infrared or ultraviolet. In such embodiments, a camera on thewearable device may be sensitive to the corresponding non-visiblespectrum and can image non-visible light displayed by these emitters.Accordingly, the totem 1200 and the wearable device can exchangeinformation via such a non-visible spectral modality. For example, thetotem 1200 and an HMD can exchange device pairing information using thenon-visible spectral modality.

The touchpad 1260 can include a force-haptic component underneath thetouch surface 1262 and a portion of the light guide 1264. Theforce-haptic component can include a haptic actuator for providinghaptic feedback to a user via the touch surface 1262 or the light guide1264. The haptic feedback can be used alone or in combination with thevisual halo to provide an indication of the user's interaction with thetotem or the objects in the user's environment.

The force-haptic component can also include a touch detector fordetecting a user's actuation of the touchpad. The force-haptic componentmay be implemented using a tough-type Alps LRA. The force-hapticcomponent can include a strain gauge with an analog-to-digital converter(ADC).

The touchpad 1260 can employ a hybrid control technique such as, e.g., atrackpoint-style hybrid velocity control input technique, a hybrid forceand velocity technique, etc. The touchpad 1260 can use the ADC to enablesuch input technique. In certain implementations, the sensitivity of thetouchpad may be less than 5 Newtons. Additional structural examples ofthe touchpad 1260 are further described with reference to FIGS. 13A and14E.

The touchpad 1260 can provide rich touch features for various userinterface experiences. For example, the user can use the touchpad 1260(alone or in conjunction with other components of the totem or thewearable system) to move or orient virtual objects. The touchpad 1260can also be used as a virtual keyboard to input texts.

Examples of a Body of the Totem

The body 1214 of the totem 1200 may be in the shape of a torch (seee.g., FIG. 12A), in which the body comprises an elongated cylinder,which may angle outward towards the top of the totem so that it is morefrustoconical in shape (which may make it easier or more comfortable tobe held in the user's hand). The body 1214 may include indentations forthe user's fingers, which may assist in gripping and holding the totem.FIG. 12B is a top view 1270 that illustrates another example of thetotem 1200. In this embodiment, the body 1214 is ovoid in shape so thatit may fit more in the palm of a user's hand or be more stable when puton a surface (e.g., a table). In some examples, the ornamentalappearance of the body 1214 of the totem 1200 may be substantially thesame as or similar to that of the totem controller, which is shown anddescribed in U.S. Design patent application Ser. No. 29/575,031, filedAug. 22, 2016, which as mentioned above is incorporated by referenceherein in its entirety. Other shapes for the body 1214 are possible suchas an elongated body having a cross-section that is oval, polygonal,etc. The cross-section of the body 1214 may be substantially symmetricso that the totem 1200 can be easily held in the user's left hand or theuser's right hand.

The body 1214 can include an upper portion 1242 which can include thetrigger 1212 and the touchpad 1260. The body 1214 can also include abottom portion 1244. The bottom portion 1244 may include an interfacesuch that the totem 1200 can be removably attached to the base 1220. Forexample, the bottom portion 1244 may be connected to the base 1220 via aconnection interface 1222 (such as, e.g., a USB-C connection) when thetotem 1200 is charging. As another example, the bottom portion 1244 maybe connected to another computing device (such as a home computer) viathe base 1220. When connected to a computer, the user can accordinglyconfigure the light placement or movement patterns of the halo (on thetouchpad 1260) via a programming interface (described below withreference to FIGS. 18A-18D) on the computing device.

The touchpad 1260 can be angled so that the touch surface 1264 and thelight guide 1262 are easily viewable by the user, when the totem 1200 isheld in the user's hand. The angle of the touchpad 1260 to thehorizontal can be in a range from about 10-60 degrees. It follows thatthe body 1214 can take on a more frustoconical shape by virtue of suchan angle. As described above, this shape may make the totem 1200 easieror more comfortable for the user to hold in their hand. In someembodiments, the angle of the touchpad 1260 to the horizontal can be ina range from about 15-30 degrees. The trigger 1212 can be disposedopposite to the touchpad 1260 so that it is easily depressible with theuser's index finger, when the totem 1200 is held in the user's hand.

In some embodiments, the bottom portion 1244 can include an illuminatedpower indicator. The illuminated power indicator may include one or moreRGB LEDs. The light placement or movement patterns of the LED powerindicator may be used to provide information related to the remainingbattery or the battery charging status for the LED totem.

The body 1214 can be configured to provide haptic feedback to a user.The body 1214 can include a haptic actuator 1760 (shown in FIGS. 17A,17B) for providing the haptic feedback. The haptic actuator 1760 may beimplemented with tough type LRA or ERM haptics. Details of thecomponents associated with the body 1214 are described with reference toFIGS. 17A, 17B.

Examples of a Home Button

The totem 1200 can include a home button 1256. The home button 1256 canbe located on the body 1214 of the totem 1200. When a user actuates thehome button 1256, the wearable system may perform various user interfaceoperations, such as, e.g., returning/opening the main menu of anapplication or the display, exiting a program, selecting an application,etc. In some embodiments, when the user actuates the home button 1256,the wearable system may enter into or wake up from a sleep mode, turn onor off the virtual content on the display, etc.

The wearable system may perform operations based on how the useractuates the home button 1256. For example, if the user actuates thehome button 1256 for an extended duration of time, the display may beturned off. As another example, if the user presses the home button 1256twice, the display may present the main page of the display while if theuser press the home button once, the display may present the main menuof the game application with which the user is currently interacting.Other functionality can be implemented by the home button 1256, e.g., aquick tap on the button can bring up an application launcher userinterface; within an application, a quick tap may bring up theapplication's menu or a long press may suspend the application and bringup the application launcher user interface.

The home button 1256 can include an RGB LED indicator which can be usedto indicate the current status of the totem or the wearable display(such as whether it is turned on/off), the battery status of the totem,the wearable display, or other components of the wearable system.

In various embodiments, the totem 1200 includes both a home button 1256and a trigger 1212 or includes only one of these user-input elements (oradditional buttons such as the bumper 1266 described with reference toFIG. 17B). The totem 1200 can be configured or shaped differently thanshown. For example, the home button 1256 may be implemented as part ofthe trigger (rather than as part of the body 1214). As yet anotherexample, the touchpad 1260 or the home button 1256 can provide visualindication for the progress of charging the totem 1200. Many variationsare possible, and the accompanying figures are intended to illustrateexample embodiments but not to limit the scope of the disclosure.

Examples of a Base

As described with reference to the body 1214 of the totem, the body 1214of the totem 1200 may be removably attached to the base 1220. Aninterface between the base and the totem can permit electrical charging,access to data in the totem or to provide software updates, etc. Forexample, the base 1220 can include a universal serial bus (USB) plugwhich can plug into a USB port on the bottom portion 1244 of the body1214. The base 1220 can further be connected to a power outlet using apower cord for charging the totem. The base can also connect with othercomputing devices such as personal computers, televisions, projectors,etc., using wired or wireless communication channels.

Examples of Actuating a Totem

The totem 1200 can support various input techniques such as, e.g.,touching, swiping, scrolling, rotary dialing, clicking, pressing, d-pademulation, etc. In some embodiments, the totem 1200 can support inertialswipes. For example, when a user swipes along a trajectory on thetouchpad 1260, a virtual object may continue moving along the trajectoryeven though the user's finger has stopped swiping. The totem 1200 canalso support edge swipes, such as, e.g., swiping on the horizontal orvertical edge of the touchpad 1260 or the light guide 1264.

A user can actuate the totem 1200 using various hand gestures and bodyposes. For example, a user can swipe or touch the touchpad 1260 or thetrigger 1212. In some embodiments (such as e.g., embodiments 1210 and1250), the totem 1200 can support 6DOF user interactions. For example,the user can rotate the totem 1200 to rotate a game avatar. As furtherdescribed with reference to FIGS. 17A, 17B, the totem 1200 can use anenvironmental sensor 1710 (such as an IMU) to detect the user'smovement. In other embodiments (such as, e.g., the example the totem1200 shown in FIG. 12B), the totem can support 3DOF (rather than 6DOF).In these embodiments, the totem can detect and translate the surge,heave, and sway movements of the users but not the pitch, yaw, and rollmovements, or vice versa.

Although the examples are described with reference to handheld devices,similar techniques may also be applied to a user input device designedfor seated and tabletop controls.

Example Structures of a Touchpad

Cross-Sectional Views of the Touchpad

FIG. 13A illustrates a cross-sectional view of an example touchpad 1260of a totem 1200. The example touchpad 1260 may be part of the totem 1200described with reference to FIGS. 12A and 12B. The touchpad 1260 caninclude 3 layers: the top layer 1312, the middle layer 1314, and thebottom layer 1316.

The bottom layer 1316 can include a printed circuit board (PCB) 1340.The PCB 1340 can mechanically support and electronically connect to oneor more components on the middle layer 1314 such as light sources 1330(e.g., LEDs) and a touch sensor 1350. The PCB 1340 can also support thearmature 1320. Additionally or alternatively, the PCB 1340 canelectronically connect other components of the totem such as, e.g.,haptic actuators (e.g., the actuators for the touchpad 1260, the trigger1212, or the body 1214, etc.), microprocessors, etc.

The middle layer 1314 of the touchpad 1260 can include a touch sensor1350. The touch sensor 1350 may be an example of the force-hapticcomponent described with reference to FIG. 12A. The touch sensor 1350can use various touch screen technologies (such as, e.g., resistive orcapacitive sensor technologies) to determine a user's actuation of thetouchpad 1260 or to provide haptic feedback. The touch sensor 1350 canbe configured to enable multi-touch technologies where it candistinguish between different levels of forces being applied to thetouch surface 1262 (and the light guide 1264). The touch sensor 1350 canprovide sensitivity in three orthogonal directions (e.g., XYZ). Exampletouch screen technologies that may be employed by the touch sensor 1350are described in FIG. 13B.

In FIG. 13A, the LEDs 1330 can be positioned in-between the touch sensor1350 and the armature 1320. The middle layer 1314 can include 6 to 12LEDs, though other numbers (e.g., 3, 15, 18, 20, etc.) of LEDs may alsobe possible. The LEDs can be positioned substantially uniformly aroundthe circumference of the light guide (see, e.g., the top views in FIGS.13E and 14A) or some LEDs may be positioned more closely to each otherthan in a uniform spacing. The LEDs 1330 may comprise a single color LED(such as, e.g., a blue-violet LED), a bi-color LED, an RGB LED (or othermultiple-color LEDs), a white LED, an organic LED (OLED), a quantum dotLED, an infrared LED, an ultraviolet LED, etc., alone or in combination.The LEDs 1330 may be controlled by a microprocessor (e.g., on the PCBboard 1340). For example, by independently adjusting each color (red,green, and blue) of an RGB LED using the microprocessor, a wide range ofcolors (e.g., a large color gamut) may be generated by the RGB LED. Thelight patterns generated by the LEDs can be associated with userinteractions with the totem or the wearable system, as well as beassociated with the objects in the user's environment. In some examples,the light patterns can also be referred to as the light placement ormovement patterns of the halo.

The armature 1320 may span the top layer 1312 and the middle layer 1314.The armature 1320 can hold the middle layer 1314 and the top layer 1312together. A portion of the armature may touch the top portion of theLEDs 1330.

The top layer can include the light guide 1264 and the touch surface1262, in addition to a portion of the armature 1320. The touch surface1262 may be surrounded by the light guide 1264 and may sit on top of thetouch sensor 1350. The user can actuate the touch surface 1262 usinghand gestures. The touch surface 1262 can also provide haptic feedbackto the user. A portion of the light guide 1264 may be overlaid on top ofthe touch sensor 1350 and the user can accordingly actuate the lightguide 1264. In some embodiments, the light guide may not be touchable(e.g., because the touch sensor 1350 is not underneath the light guide1264). However, the light guide may still provide visual or hapticfeedback.

The light guide 1264 may be in-between the armature 1320 and the touchsurface 1262 on the top layer 1312. The light guide 1264 may include afirst portion 1362 and a second portion 1364. The first portion 1362 ofthe light guide 1264 can be overlaid on top of the LEDs 1330. The lightguide 1264 can diffuse the light generated by the LEDs 1330 so as tospread the light out over an extent of the light guide 1264 that islarger than the individual LED packages, which may provide a morepleasing visual appearance. The light patterns can provide visualfeedback associated with the user's interactions with the wearablesystem or the touchpad 1260. The light patterns can also provideinformation associated with objects in the user's environment (such asthe relative positions between a user and an object).

The second portion 1364 of the light guide 1264 may be positioned overthe touch sensor 1350. For example, the light guide 1264 may bepositioned with a 0.75 mm to 2 mm inset over the touch sensor 1350.Because a portion of the touch sensor 1350 can extend under the lightguide 1264, the light guide 1264 can not only provide visual and hapticsfeedback but also can be actuated by the user. As shown by the finger1310, the user can actuate the light guide 1264, for example, bytouching, tapping, swiping, or pressing the light guide 1264.Accordingly, when the user actuates the light guide 1264, the totem cansimulate user interface experiences such as, e.g., tapping onselectively illuminated “buttons” or “arcs” of light on the light guide.

The light guide 1264 does not have to be positioned over the touchsensor 1350. As a result, the user may not be able to actuate the lightguide 1264 to interact with the totem or the wearable device. In someimplementations, although the second portion of the 1364 is positionedover the touch sensor 1350, the processor 1770 or the local processing &data module 260 may be configured not to recognize the user's input onthe light guide 1264. As a result, the light guide 1264 may be used toprovide visual or haptic feedback but may not be responsive when a useractuates the light guide 1264.

FIG. 13B illustrates examples of touch screen technologies. In theseexamples, the touch sensor 1350 (e.g., 1350 a, 1350 b, and 1350) issandwiched in-between the touch surface 1262 and the PCB 1340. Theembodiment 1362 of the touch screen technology illustrates an example ofthe force under touch technology. In this example, the touch sensor 1350a can comprise a pressure sensitive sensor which can detect multiplelevels of force exerted on the touch surface 1262. In certainembodiments, the touch sensor 1350 can also be configured to relayhaptic feedback to the touch surface 1262.

In the example 1364, the touch sensor 1350 b may comprise strain gauges.The strain gauges may be located underneath the edge of the touchsurface 1262. When a user actuates the touch surface 1262, the touchsensor 1350 b can determine the user's gesture (e.g., a press or a tapgesture) by measuring the strains of the touch surface 1262.

The example 1366 illustrates an interpolating force-sensitive resistancetouch sensor technology (IFSR). The touch sensor 1350 c can use forcesensitive resistors whose level of sensitivity can change depending onthe level of pressure. Accordingly, the touch sensor 1350 c may detectmultiple levels of force exerted on the touch surface 1262.

FIGS. 13C and 13D illustrate additional cross-sectional views of anexample touchpad of a totem. The cross-sectional view 1380 shows a topcover 1381 a and a top housing 1381 b underneath the top cover 1381 a.Either the top cover 1381 a or the top housing 1381 b, or thecombination may be part of the armature 1320 shown in FIG. 13A. The tophousing may be attached to a light guide 1264 (e.g., light pipe 1381 c)with the adhesive. The light pipe can be transparent or translucent. Invarious embodiments, the light pipe 1381 c can comprise diffuser sheets,diffuser films, an etched waveguide, a transmissive optical elementcomprising layers of particles, irregular surfaces, holographs, whitesurfaces, ground glass, polytetrafluoroethylene (PTFE or Teflon), opalglass, greyed glass, colored gel, and so forth. Embodiments of a lightpipe comprising optically diffusive material advantageously may diffuseand spread out the light from the LEDs 1383 a so that the light outputin the direction 1384 appears to have an overall, substantiallycontinuous glow (when all LEDs are illuminated) rather than appearing asdiscrete, individual LEDs (if the light pipe were substantiallytransparent).

The top housing 1381 b can also be attached to a white Mylar 1381 c (orother reflective sheeting) using the adhesive. The top housing canenclose one or more LEDs 1383 a. For example, the top housing mayinclude an RGB LED which includes a package of a red LED, a green LED,and a blue LED. Therefore, if the touchpad includes 12 RGB LEDs, thetouchpad may include 12 top housing structures each includes an RGB LED.As another example, the top housing may include all 12 RGB LEDs for thetouchpad.

The top cover is adjacent to the touch cover 1383 d. The touch cover maybe an embodiment of the touch surface 1262 shown in FIG. 13A. The touchcover may be attached to the touch board 1383 using touch cover adhesive1383 c. The touch board include an embodiment of the force-hapticcomponent described with reference to FIG. 12A. As further describedwith reference to FIG. 13D, an adhesive stack 1386 may be sandwichedin-between a portion of the touch board and a portion of the tophousing. The adhesive stack 1386 may be used to fixatedly attach theportion of the touch board with the top portion of the top housing.

The adhesive stack 1386 may include two layers: a shock-absorbing foampad (e.g., Poron, available from the Rogers Corp. Rogers, Conn.), and aflex and adhesive layer. The top portion of the poron pad 1387 a may beattached to the bottom portion of the touch board and the bottom portionof the poron pad may be attached to the flex and adhesive layer 1387 b.The flex and adhesive layer may be attached to the top housing. Theporon pad may be composed of two 0.2 mm thick and one 1.00 mm thickadhesives. The one 1.00 mm adhesive may be positioned in-between the two0.2 mm thick adhesives. Alternatively, one of the two 0.2 mm thickadhesives may be in-between the 1.00 mm thick adhesive and the other 0.2mm thick adhesive.

With reference back to FIG. 13C, one or more LEDs 1383 a may be attachedto the touch board. The LED can emit light in the direction 1382. Thelight may reach the light pipe, which can transmit the light in thelight output direction 1384 through total internal reflection (TIR). Thebottom portion of the light pipe can also be attached to a portion ofthe white Mylar. The white Mylar (or other reflective sheeting) mayreflect a portion of the light emitted by the LED to increase the amountof light output in the light output direction 1384. The light pipe maydiffuse light from the LED so that the light output appears as more of aangularly continuous output rather than individual, discrete lightsources.

Bottom Views of the Touchpad

FIG. 13E illustrates a bottom view of an example touchpad. The touchpad1390 may be an embodiment of the touchpad 1260. The touchpad 1390 caninclude 12 LEDs 1394 a, though other numbers of LEDs are also possible.The 12 LEDs 1394 a are equally spaced circumferentially in this example,although in other examples (such as the examples described withreference to FIGS. 14B and 15 ), the spacing among the LEDs may notnecessarily be equal. The 12 LEDs may be located on the bottom of thetouchpad. The touchpad can selectively illuminate the LEDs to create alight pattern which may appear as a halo around the touch pad.

The LEDs can be attached to the housing 1392. The housing can have twoalignment pin holes and three snap clearances 1394 e. The two alignmentpin holes may be on the opposite end of the diameter of the touchpad.The three snap clearances may be spaced equally. The alignment pin holes1394 d and the snap clearances 1394 e may be used to orient and positionthe touchpad in relation to the totem body. The snap clearances 1394 emay also be used to fix the touchpad 1390 to the totem body.

The touchpad 1390 can employ a touch screen technology such as, e.g., aforce flex stack 1394 b. A portion of the force flex stack 1394 b may beoverlaid on top of the touch board component area 1394 c. The force flexstack 1394 b may include the touch surface 1262, alone or in combinationwith the touch sensor 1350. The force flex stack can detect user inputon the touchpad 1390, such as, e.g., pressing, swiping, tapping,touching, etc.

In certain embodiments, the touch board component area may include thePCB 1340. The touch board component area can include a force-hapticcomponent (such as, e.g., a haptic actuator), a microprocessor, one ormore communication units (such as, e.g., an ADC), etc.

Top View of the Touchpad

FIG. 14A illustrates a top view of an example touchpad of a totem. Thetouchpad 1260 in FIG. 14A can include the armature 1320, the light guide1264, and the touch surface 1262 (which may further be broken down intointeractable regions 1262 a and 1262 b). The light guide 1264 may bepositioned on top of the light sources (e.g., LEDs) 1330 (shown indashed lines in FIG. 14A).

Advantageously, in this embodiment, the touchpad 1260 can be dividedinto three interactable regions: the light guide 1264, the first portionof the touch surface 1262 a, and the second portion of the touch surface1262 b. As described with reference to FIGS. 12 and 13A, eachinteractable region may be mapped to a type of user interaction. Forexample, the user can touch the light guide 1264 to moveforward/backward on a browser while swipe on second portion the touchsurface 1262 b to move the web content upward/downward. The user canalso swipe in a circular motion on the first portion of the touchsurface 1262 a to move a virtual object closer (or farther) away fromthe user.

Although this example includes three interactable regions, the totem mayinclude more or less interactable regions in other embodiments. Forexample, rather than dividing the touch surface into two interactableregions (corresponding to the first portion 1262 a and the secondportion 1262 b), the touch surface may be divided into 4 interactableregions with each region occupies a quadrant of the touch surface. Thetouch surface may also include only one intractable region. Additionallyor alternatively, the light guide 1264 may not be interactable, becausefor example the processor 1770 or another module of the wearable systemmay not be programmed to recognize the user input from the light guide1264 or the touch sensor 1350 does not extend underneath the light guide1264.

The type of interactions mapped to a touch region may also be customizedbased on the type of user interactions. For example, when a user iswatching a television, the touchpad may simulate a 4 way d-pad (up,down, left, right) on the light guide 1264. But if the user is browsinga webpage, the light guide 1264 may support backward/forward userinterface operations.

The interactable regions can also provide visual or haptic feedback tothe user. For example, the touch surface 1262 can be configured toprovide haptic feedback while the light guide 1264 can provide bothhaptic feedback and visual feedback (e.g., via the light placement ormovement patterns of the LED halo).

Layout of Light Sources

FIG. 14B illustrates an overview of an example layout of light sources(e.g., LEDs) associated with the touchpad. FIG. 14B shows a touchpad1450 which has a visible circular surface 1452 with 12 LEDs (illustratedas LED 0, LED 2 . . . LED 11) on the perimeter. These 12 LEDs can beused to represent specific information when they are lit in a specificsequence. For example, each LED may be assigned to a bit thus resultingin a 12 bit word. As described with reference to FIGS. 24B, 20A, and20B, the outward-facing imaging system 464 of a wearable device cancapture the display sequence of the LEDs. The wearable device can thenanalyze the images and to extract information from the display sequence.The extracted information may be used, for example, to pair the totemwith the wearable device, calibrate the totem, etc. For example, whenLEDs 0, 3, 6, 9 are illuminated, the wearable device may recognize thata pairing between the wearable device and the totem is initiated.Accordingly, the wearable device may search for wireless (e.g.,Bluetooth) devices in its environment to determine whether there are anynew devices (such as the totem associated with the LEDs). When the LEDsare multi-colored LEDs (such as e.g., RGB LEDs), the amount ofinformation represented the LED light patterns may increase (forexample, to be more than the 12 bit word). For example, the blue lightat the LEDs 6, 7, 8 may indicate that the totem is in a pairing modewhile the yellow light at the LEDs 6, 7, 8 may indicate that the totemis in a charging mode.

In some embodiments, the LEDs may be divided (e.g., multiplexed) intomultiple groups that are separately illuminated. Each group may be usedto represent specific information. FIG. 14B shows four example groups(illustrated as MUX 0 (1454 a), MUX 1 (1454 b), MUX 2 (1454 c), and MUX3 (1454 d)), with each group including three LEDs. For example, when theLEDs of MUX 1 are illuminated (e.g., the LEDs 3, 4, 5), the wearabledevice may recognize that the totem is currently indicating the totem'supward movement, whereas when the LEDs of MUX0 are illuminated, thewearable device may recognize that the totem is rotated to the left, andso forth.

In other embodiments, a different number or layout of light sources ormultiplexed groups can be utilized and a different amount of information(e.g., a word length different from 12 bits) may be represented byilluminated groups or sequences of LEDs.

As previously described with reference to FIG. 12A, the light source canemit light in the non-visible spectrum (e.g., infrared or ultravioletlight). Such light can be captured by a wearable device and the lightpattern in the non-visible spectrum can also be used to convey theinformation from the totem to the wearable device. The information mayinclude the totem's device information (e.g., for pairing the totem withthe wearable device), or other types of information, such as, e.g., thetotem's status (e.g., whether it is low in battery, whether theconnection is established, devices that are currently paired with thetotem, etc.).

FIG. 15 illustrates example LED layouts or patterns of light from an LEDlayout. In this example, the LEDs are placed cardinally, e.g., withreference to a North-East-South-West (NESW) coordinate system 1592, inwhich North may point away from the user, South toward the user, Easttoward the user's right, and West toward the user's left, when the totemis held in the user's hand. The coordinate system 1592 may be rotated bya certain degree. For example, the layouts in the column 1550 arecardinally placed with reference to a neutral position, where thecoordinate system 1592 is not rotated. In contrast, the layouts incolumns 1570 and 1580 are rotated 15 degrees clockwise from the cardinalposition shown in the NESW coordinate system 1592.

The LED layouts may be user adjustable. For example, the touchpad (orthe light guide) on the totem may be mechanically rotatable. As anexample, the LED layout of a totem may initially include a 15 degreerotation clockwise for a right-handed user. However, when a left-handeduser uses the same totem, the left-handed user can rotate the touchpad30 degrees counter-clockwise to allow better interactions the touchpadusing his or her thumb. The example layouts in the columns 1560 and 1580can be composed of 12 LEDs. These LEDs can be placed adjacent to eachother. For example, the 12 LEDs can be placed next to each other withoutspacing to form a circular layout as illustrated in pattern I.

In some embodiments, two neighboring LEDs may be placed with a spacein-between (as illustrated by patterns in the columns 1550 and 1570).The space may roughly be the size of one LED. For example, the pattern Dshown in column 1550 and row 1540 may include 6 LEDs where they areplaced roughly in the same distance part from each other. However, insome situations, to produce the pattern in the columns 1550 and 1570,the totem may selectively turn off or does not include one or more LEDs.For example, the pattern D may include 12 LEDs (rather than 6 LEDs) and6 out of the 12 LEDs may not be illuminated in pattern D. In contrast,if the user changes the user interface operations, more or fewer thanthe 6 LEDs as shown in pattern D may be illuminated. Advantageously, insome embodiments, the totem can save on battery consumption becausefewer LEDs are used (with spacing between them) or because the LEDs canbe selectively turned off to achieve a desired illumination pattern(such as those shown in FIG. 15 ). In addition, by spacing theilluminated LEDs apart from each other (either physically or by notilluminating some intermediate LEDs), the user may not need to move hisor her thumb to a precise location in order to perform a certain userinterface operation, which can reduce user fatigue in some situations.

The LEDs may be divided into multiple groups (see, also, the examplemultiplexed groups described with reference to FIG. 14B). Rows 1510,1520, 1530, and 1540 of FIG. 15 illustrate examples ways of grouping theLEDs. For example, layouts A and G in row 1510 include 2 groups of LEDswith each group being represented by a different color. Layouts B and Hin row 1520 include 3 groups; layouts C, D, and F in row 1530 include 4groups, and layouts D and I include 6 groups. Each group of LEDs mayhave similar characteristics, such as, e.g., similar light patterns,similar reactions to user interactions (e.g., all light up at the sametime), etc. Additionally or alternatively, as described with referenceto FIG. 14B, each group of LEDs may be used to represent specificinformation. For example, a wearable device may determine that the usercan entered into a video recording mode when its outward-facing imagingsystem captures that one or more LEDs associated with the top arc inpattern B lights up.

Advantageously the LED layouts may be aligned to the grip direction ofthe totem. Layouts A-I (highlighted in black) are such examples. Inthese layouts, when the user grabs the totem, the user can naturallyreach the groups of the LEDs without significant adjustments of his orher hand.

In certain implementations, only a portion of the LEDs isuser-interactable. An LED may be interactable if a user can actuate theLED (or the light guide associated with the LED). For example, pattern Eincludes 4 LED groups: the top arc, the bottom arc, the left arc, andthe right arc, with each group having 3 LEDs. The user may be able toactuate the top and bottom arcs (e.g., by depressing the light guideabove these arcs), but not the left and right arcs. However, the leftand right arcs can still provide visual or haptic feedback alone or incombination with the top and bottom arcs. Similar considerations applyto the other layouts shown in FIG. 15 .

In addition to or in alternative physical positions of the LEDs, thepatterns in FIG. 15 can be light patterns illuminated by the LEDs. Forexample, the totem can present the pattern E by illuminating a ring of12 LEDs while present the pattern D by illuminating every other LEDs inthe ring of 12 LEDs. As another example, rather than user rotating thetouchpad or the light guide, the totem can be programmed to adjust theplacement of the halo based on whether the user is left-handed orright-handed. The totem may display an arc at the 1 o'clock position fora left-handed user but display the same arc the 11 o'clock position fora right-handed user. The arc may be associated with a user interfaceoperation, such as, e.g., moving a webpage upward.

Although many embodiments of the user input device are described interms of the totem 1200, this is for illustration and is not alimitation on the types of user input device usable with the wearablesystem 200. For example, in other embodiments, a display screen (orvariant thereof) may be used to display the illumination shapes andpatterns to a user. For instance, in some of these embodiments, a smartphone may be leveraged as a totem and may display various shapes andpatterns on its screen. In some examples, the totem 1200 can include adisplay screen, which may be touch sensitive (similar to the surface ofa smart phone). In some such embodiments comprising a display screen,the use of an illuminated light guide 1264 may be optional, since thedisplay screen can be used to display the illumination patterns andtransition sequences described herein.

Examples of Placement and Movement of Light Patterns

FIGS. 16A and 16B illustrate example placement or movement patterns oflight emissions from a halo of a totem (or other type of light-emittinguser input device). As described above, the halo may comprise anilluminated portion of the totem surrounding the touch pad region. Asillustrated in the embodiments shown in FIGS. 16A and 16B, the lightguide substantially diffuses the light emitted by individual lightsources so that the halo appears as a ring or an arc-like portion of aring rather than individual, discrete light sources.

The placement and movement of light patterns that can be projected bythe halo can comprise one or more characteristics such as, e.g., shape,color, brightness, position, size, movement, animation, other visualeffects (e.g., blink or flash, fade in, fade out), etc. For example,FIG. 16A illustrates four example patterns. In the pattern 1612, thehalo is shown as a light blue color with a bright pink color at the 2o'clock position. The halo in pattern 1614 has 4 colored arcs: yellow,green, blue, and red, which correspond to the top, bottom, left, andright positions (respectively) of the touchpad. The pattern 1616 shows adark blue halo and the pattern 1618 shows a dark blue halo with a lightblue arc on the top.

The light placement or movement patterns of the halo can provide visualfeedback to the user and persons in the user's environment. The lightplacement or movement patterns may be determined based on contextualinformation, such as, e.g., a user's environment, a user'scharacteristics, information associated with an object, a process, acomponent of the wearable system, a user's interaction with the wearablesystem, etc.

As an example of generating a halo based on the user's environment, anenvironmental sensor 1710 (described below with reference to FIGS. 17A,17B) in the totem 1200 can be a light sensor which can detect whetherthe user is in a light or dark environment. If the user is in a darkenvironment, the totem 1200 can display a halo with a brighter color(such as a white color) to help user to perceive the halo. On the otherhand, if the user is in a bright environment, the totem 1200 may displaya darker color to help the user to distinguish the halo from ambientlight in the user's environment. As another example, in a dark room theintensity of the light in the halo may be decreased (e.g., a “nightmode”), because it is easy to perceive the illuminated halo in a darkenvironment. Conversely, in a bright environment (e.g., outside insunlight), the intensity of light in the halo may be increased, so thatthe halo patterns are visible in the bright environment.

The patterns in the halo can also be based on the user'scharacteristics, such as, e.g., the user's physiological data,demographic information (e.g., age, location, occupation, preferences,etc.), and so forth. For example, when the user is playing a racinggame, the totem can display a halo corresponding to the user's directionof driving. The color of the halo may be red initially. However, whenthe wearable system (such as, e.g., the environmental sensor on thetotem or the HMD) detects that the user's heart rate or respiratory rateexceeds a threshold condition, the wearable system may determine thatthe user is in an emotionally intense state. Accordingly, to soothe theuser's emotional state, the wearable system may change the color of thehalo from red to a blue (such as the pattern 1618, where the light bluearch illustrates the direction of driving).

Information associated with an object can include, for example, anotification or an alert associated with the object, characteristics ofthe object (such as, e.g., a function, a type, a location, a shape, anorientation, etc.), and so forth. As an example, the object may be avirtual email application that is displayed to the user by the HMD. Anew email received by the virtual email application may be indicated tothe user by a halo having an iridescent light pattern on the totem. Asanother example, the halo may be an indication of whether an object isperceivable in the user's environment. For example, in a treasure huntgame, the halo may correspond to a location of the treasure. If thetreasure is located in the front right position of the user, the totemmay display a halo at the 1 o'clock position of the light guide (seee.g., pattern 1612). As yet another example, if an object is notinteractable by the user, the wearable system may be configured not todisplay the halo.

As described with reference to FIGS. 20A-21E, the halo can also be usedto guide a process or to indicate the status of a process. For example,the wearable system can apply a computer vision algorithm to analyze theplacement of the halo for pairing the totem with another device. Thewearable system can also analyze appearance of the light patterns usingthe computer vision algorithm to calibrate the totem in a user'senvironment. When the pairing or the calibration status is complete, thetotem may display the pattern 1616 to indicate to the user that theprocess has been completed.

The halo can further be used to indicate information associated with acomponent of the wearable system. For example, the totem may be used tolocate a battery pack that is used to provide power to the wearabledevice. In this example, the battery pack may include anelectro-magnetic emitter while the totem may include an electro-magneticsensor. The electro-magnetic sensor can detect the magnetic fieldgenerated by the electro-magnetic emitter and calculate the relativepositions between the totem and the battery pack accordingly. The haloas displayed by the totem can correspond to the relative positionsbetween the totem and the battery pack (to help the user to find thebattery pack). For example, a halo with a large arc at the 3 o'clockposition may indicate that the battery pack is near the user and to theuser's right. However, a halo with a small arc at the 3'clock positionmay indicate that the battery pack is far away from the user but it isstill to the user's right. As another example, when the power of thewearable device is low, the totem may display a halo with a small arcindicating that the battery is about to die. Although described in termsof a battery pack, any component of the wearable system or otherwise(e.g., a user's car keys) can include an electromagnetic emitter thatcan be tracked by the totem, which can illuminate the halo to assist theuser in finding the component.

The halo can also provide an indication of a user's interaction with thewearable system. Advantageously, this indication may inform a person inthe user's environment of the user's current interactions. For example,when a user is recording a video, the halo may be flickering in red.Accordingly, a person next to the user will be able to see thisflickering red light pattern and know that he or she should notinterrupt the user's video recording experience. As another example,when a user wins a level in a game, the halo might light up with thesame colors as the level of the game to show to the user's friends thatthe user has passed the level.

In addition to or in alternative to visual feedback, the halo can guideuser interactions. For example, the halo may show 4 arcs (top, bottom,left, and right) with different colors (as illustrated by the effect1614) to indicate that the user can use the touchpad of the totem as ad-pad. As another example, while a user is watching a television programusing the display 220, the totem may present the halo pattern 1662 inFIG. 16B. In the pattern 1662 includes a red arc and a blue arc. The arclength for the red arc and the blue arc may be approximately 0.25π.However, when the user actuates the totem (such as by tapping the touchsurface of the totem), the totem may present a fade-in effect (asillustrated by process 1 in FIG. 16B) of the arcs. As a result of thefade-in effect, the length of the red arc and the blue arc may beincreased from approximately 0.25π to 0.75π as shown in the pattern1664. The brightness of the red and blue arcs may also increase due tothe fade-in effect. This increased brightness and arc lengths mayprovide user with a clearer indication that the user can swipe leftwardor rightward on the touch surface. But when the totem detects that theuser has not interacted with the totem for a threshold duration of time.The totem may present the fade-out effect as illustrated by process 2 inFIG. 16B. Accordingly, the halo is changed from the pattern 1664 to1662. As a result, the size of the arcs and the brightness of the arcswill decrease. Advantageously, by reducing the size of the arcs and thebrightness of the arcs, the totem may reduce the battery consumptionwhen the user is not interacting with the totem.

In some embodiments, the totem 1200 can present the halo in conjunctionwith haptic, audio, visual, or other effects to provide informationassociated with contextual information or to guide a user's interaction.

Other Components of the Totem

FIGS. 17A and 17B illustrate examples of components of a totem 1200. Theexample totem 1200 can include a touchpad 1260 (which can include atouch surface 1262 and a light guide 1264), a trigger 1212, and a totembody 1214 as described with reference to FIGS. 12A and 12B. The lightguide 1264 may comprise user-interactable regions (e.g., touchsensitive) and may at least partially or completely surround the touchsurface 1262. The totem 1200 can also include a variety of components,at least some of which may be disposed inside the body 1214 of thetotem. These components will be described further below and can includean environmental sensor 1710, a battery 1720, a near-field communication(NFC) interface 1732, a network interface 1734, a haptic actuator 1760,and a processor 1770. A connection interface 1222 can be disposed at thebottom of the body 1214 to, for example, removably attach the totem 1200to a base. The connection interface 1222 can be used to provideelectrical power to charge the battery 1720 and to provide acommunication link between the components of the totem and externaldevices (e.g., a computing device).

Bumper

The totem 1200 can include a button 1266 (referred to as a bumper) thatin the example shown in FIG. 17B is located at the front end of thetotem, above the trigger 1212 and below the touchpad 1260. The bumper1266 may provide an ergonomically comfortable place for the user to resthis or her forefinger. The bumper 1266 can comprise a touch sensitivesurface implemented as, e.g., a depressible button, a capacitive touchsensor, a force-haptic element, etc.

In the example shown in FIG. 17B, the user can actuate the totem 1200primarily using three fingers, e.g., the thumb to actuate the homebutton 1256 or the touchpad 1260, the forefinger to actuate the bumper1266, and the middle finger to actuate the trigger 1212. Such athree-finger actuatable totem can permit the user to rapidly andefficiently provide user input without excessive and tiring use of justone finger (as may occur with mouse setups for desktop computers). Thedifferent buttons 1212, 1256, 1266 can provide different functionality,e.g., depending on whether a user is working within an application,scrolling through an application launcher, selecting objects in theenvironment, etc. In some cases, a user may use just the forefinger toswitch back and forth between pressing the bumper or pulling thetrigger.

As examples of bumper functionality, tapping the bumper while the useris looking at an application can bring up an options menu for thatapplication, while long pressing the bumper can activate a manipulationsequence for virtual objects in the environment. For example, a longhold of the bumper can grab an object, and a long press on the bumperwhile pointing the totem 1200 toward the object can activate directmanipulation of the object (e.g., to move or re-orient the object).Tapping the bumper (or pulling the trigger) in a manipulation sequencecan end the sequence.

Environmental Sensors

An environmental sensor 1710 can be configured to detect objects,stimuli, people, animals, locations, or other aspects of the environmentaround the user. The environmental sensors may include image capturedevices (e.g., cameras), microphones, IMUs, accelerometers, compasses,global positioning system (GPS) units, radio devices, gyroscopes,altimeters, barometers, chemical sensors, humidity sensors, temperaturesensors, external microphones, light sensors (e.g., light meters),timing devices (e.g., clocks or calendars), or any combination orsubcombination thereof. In some embodiments, the environmental sensorsmay also include a variety of physiological sensors. These sensors canmeasure or estimate the user's physiological parameters such as heartrate, respiratory rate, galvanic skin response, blood pressure,encephalographic state, and so on. Environmental sensors may furtherinclude emissions devices configured to receive signals such as laser,visible light, invisible wavelengths of light, or sound (e.g., audiblesound, ultrasound, or other frequencies). In some embodiments, one ormore environmental sensors (e.g., cameras or light sensors) may beconfigured to measure the ambient light (e.g., luminance) of theenvironment (e.g., to capture the lighting conditions of theenvironment). Physical contact sensors, such as touch sensors, straingauges, curb feelers, or the like, may also be included as environmentalsensors.

The information acquired by the environmental sensor 1710 may be used todetermine the light placement or movement patterns of the halo displayedon the totem. For example, the environmental sensor can determine therelative positions between the user and a physical object in the user'senvironment using a GPS sensor or an electro-magnetic sensor (fordetecting an electromagnetic signal associated with the physicalobject). The totem may present a halo whose placement can correspond tothe location of the physical object. For example, if the object is infront of the user, the totem can present a halo in the 12 o'clockdirection on the light guide 1264.

Additionally or alternatively, the information acquired by theenvironmental sensor 1710 can be used for one or more user interfaceoperations. For example, the totem 1200 can detect the totem's 6DOFmovement using the IMUs. For example, when the user rotates the totem1200 while playing a game, an avatar (or other virtual object)controlled by the user (and displayed to the user via a wearable device)can rotate accordingly based on the movement data acquired by the IMUs.Additionally or alternatively to moving or rotating the totem 1200, theuser can provide input on the touchpad 1260. For example, movement ofthe user's thumb toward or away from the user (e.g., along the long axisof the totem) can move the virtual object toward or away from the user.Movement of the user's thumb in a transverse direction back and forth onthe touchpad 1260 can scale the size of the virtual object (e.g., fromlarger to smaller or vice versa) or rotation of the user's thumb aroundthe touchpad can rotate the virtual object.

Although in this example, the environmental sensor is located on thetotem 1200, in some embodiments, the environmental sensor may be locatedat other components of the wearable system described herein. Forexample, an environmental sensor (such as a camera or a physiologicalsensor) may be part of the display 220 of the wearable system 200 (shownin FIG. 2 ).

Battery

The battery 1720 stores electric power for the totem 1200. The totem candetermine the current status of the battery 1720 using the processor1770. For example, the processor 1770 can measure and calculate theamount of power left in the battery 1720, whether the battery 1720 iscurrently being used, and the remaining life of the battery 1720 (suchas when the battery 1720 will need to be replaced). As further describedin FIGS. 21A-21E, the current status of the battery 1720 may bevisualized via visual feedback presented on the light guide 1264, thehome button 1256, an LED power indicator located on the bottom portion1244 (e.g., shown in FIG. 12A) of the totem body 1214, or the display.

The totem 1200 can also determine the power consumption level of thetotem using the processor 1770. For example, the totem can estimate thepower consumption based on the estimated current required to output acertain light signature. The light signature may include a certain lightplacement or movement pattern of a halo associated with an indication ofa process, a status, an object, or a user interaction. As an example ofestimated electric currents associated with light signatures, the totem1200 may require 5.6 mA to perform a 3 or 6 DOF calibration; 48 mA tofind an object in the user's environment or to indicate a direction; and54 mA to perform a wireless (e.g., Bluetooth) pairing. As anotherexample of estimated currents for light signatures, 14.85 mA of thecurrent may be consumed to provide an indication that the battery isless than 15%; 21.6 mA may be used to indicate that the totem is in asleep mode or in an idle state; and 72.4 mA of current may be suppliedto provide incoming notifications (such as, e.g., of new messages).

These estimated electric currents are merely examples. The specificamount of electric power needed may depend on various factors such asthe type of LEDs used, the color of the LED, the placement and movementlight patterns of a halo, etc. For example, the totem battery 1720 maydrain 2 hours faster if every RGB LED of the touchpad 1260 is set todisplay white at 100% brightness all the time rather than displayinggreen at 100% brightness. This is because when an RGB LED displaysgreen, only ⅓ of the LED (e.g., the green LED in the RGB LED) isutilized but when an RGB LED displays white, all LEDs in the RGB LED)are utilized.

To reduce the overall power consumption, the LEDs (particularly the LEDsthat are interactable) may be spaced apart so as to reduce the totalnumber of LEDs on the touchpad. For example, the totem may employ thelayouts shown in the columns 1550 and 1570 in FIG. 15 . Advantageously,in some embodiments, by spacing the LEDs apart, the power consumptionmay be reduced by up to 50%. The specific amount of reduction of thepower consumption may depend on the layout of the LEDs. For example, thepattern D in FIG. 15 may have lower power consumption than the pattern Abecause pattern D has fewer LEDs being illuminated. As another examplefor reducing the power consumption, the totem can run the LEDs with lessbrightness (such as, e.g., at <40% brightness).

The totem can manage the power consumption based on the objects that theuser is interacting. The totem can turn off the LEDs that do not have anactive user interface option. For example, the totem may simulate ad-pad using LED layout E shown in FIG. 15 . However, during a game, thedownward button may be disabled at a certain level. Accordingly, thetotem may be configured not to display the bottom arc. Additionally oralternatively, the totem can manage the power consumption based on thestatus of the totem, the display, or other components of the wearablesystem. For example, the totem can turn the halo off after detecting acertain amount of inactivity. The totem can make such detection bycalculating the amount of movement data of the totem acquired by theIMU.

In some embodiments, the wearable system may include a battery packwhich may be attached to the user (such as, e.g., at the user's waist).The battery pack may be connected to the totem or the wearable device tosupply power. The battery 1720 may be part of the battery pack or may beused in connection with the battery pack to supply electricity to thetotem 1200.

Near-Field Communication (NFC) Interface and Network Interface

The NFC interface 1732 and the network interface 1734 can be configuredto allow the totem to pair or communicate with a target object, such ase.g., another totem (or other user input device), a wearable display, apersonal computer, a headphone, a key, a server, or another physicalobject. The NFC interface 1732 can be used for short-range wirelesscommunications (such as when totem is in 10 cm or less distance from thetarget object). In some embodiments, the NFC employs electromagneticinduction between two loop antennas when NFC-enabled devices—for examplea totem and an HMD—are positioned close to each other (within about 10cm) and wirelessly exchange information. This exchange of informationmay be operated within the globally available unlicensed radio frequencyISM band of 13.56 MHz over an ISO/IEC 18000-3 air interface at ratesranging from 106 to 424 Kbit/s. A variety of communications protocolsand data exchange formats may be used for the NFC. Some non-limitingexamples include: ISO/IEC 14443 and Felicia, ISO/ICE 18092, thestandards defined by the NFC Forum and the GSMA group, etc.

The network interface 1734 can be configured to establish a connectionand communicate with the target object via a network. The network may bea LAN, a WAN, a peer-to-peer network, radio frequency, Bluetooth, Wi-Fi,a cloud based network, or any other type of communication network.

For example, the totem and the wearable device may be paired using awireless communication protocol such as, e.g., Bluetooth Low Energy(BLE). BLE may be advantageous in some embodiments because it canmaintain bandwidth during BLE audio streaming. As further described withreference to FIGS. 20A-20C, various techniques may be used to pair thewearable device with the totem. For example, the wearable system candetect a certain light pattern presented by the totem by analyzingimages of the totem obtained by an outward-facing camera. The detectionof the light pattern can trigger the wearable system to start thepairing process. During the pairing process, the user may interact withthe wearable display via a dialog box or user interface (UI) wizard tospecify the settings and parameters associated with the totem, thewearable display, the type of information shared between the totem andthe wearable display, the type of communication channel, etc.

Although in this example, the NFC interface 1732 and the networkinterface 1734 are illustrated as separate components, in someembodiments, the NFC interface 1732 and the network interface 1734 maybe part of the same communication device or system.

Haptic Actuator

The totem 1200 can include a haptic actuator 1760. As described withreference to FIG. 12A, the haptic actuator 1760 can provide a hapticfeedback to the user. One or more haptic actuators 1760 may beimplemented in the totem 1200. A haptic actuator may be located at thetrigger 1212, the touchpad 1260, or the totem body 1214.

The haptic feedback may be provided based on the user's interactions,the contextual information associated with an object or the user, statusof the totem or other components of the wearable device, etc.

Processor

The hardware processor 1770 can be configured to process data acquiredby the environmental sensor 1710. The processor 1770 can also receivefrom and send data to another device (such as the wearable display oranother paired device) via the NFC interface 1732 or the networkinterface 1734. The processor 1770 can analyze these data to determinethe placement or movement patterns of the halo at a given time. In someembodiments, the processor 1770 may work in conjunction with anothercomputing device (such as, e.g., the wearable device, a remote server,or a personal computer) to analyze. For example, the processor 1770 maydetect the user's movement using the environmental sensor 1710. Theprocessor 1770 can pass the user's movement data to the local processingand data module 260 or the remote processing module 270 for furtheranalysis. The processor 1770 can receive the result of the analysis fromthe local processing and data module 260 or the remote processing module270. For example, the processor 1770 can receive information on theposition and movement trajectory of a competitor in a game. Based onthis information, the processor 1770 can instruct the LEDs emit lightsfor displaying a halo corresponding to the position and movementtrajectory of the competitor.

The hardware processor 1770 can process various user inputs from theuser's actuation of the totem 1200. For example, the hardware processor1770 can process user inputs on the touchpad 1260, the trigger 1212, thehome button 1256, the bumper 1266, etc., as described with reference toFIGS. 12A-17B. As an example, the hardware processor can detect a user'shand gesture on the touchpad 1260 by processing signals from the touchsensors. As described herein, the hardware processor 1770 can processthe user's input and instruct other components of the totem 1200 (suchas, e.g., the LEDs or the actuator 1760) to provide an indication of theuser's actuation of the totem 1200.

Connection Interface

As described with reference to FIG. 12A, the connection interface 1222may be used to establish a connection with another device. Theconnection may be a wired connection. For example, the connectioninterface 1222 may comprise a USB connector. The connection interface1222 may also comprise a USB port, such as, e.g., a USB-B type port,USB-A type port, a micro USB, or USB Type C port, etc. The connectioninterface 1222 can also include a power cord for connecting to a powersource for charging.

The example components illustrated in FIGS. 17A and 17B are not limitingexamples, and the totem 1200 can include fewer or more or differentcomponents than illustrated. For example, the totem 1200 may not have anNFC interface 1732. As another example, the totem 1200 can include anoptical emitter configured to emit a light (such as an infrared light)or an electro-magnetic emitter configured to generate or sense amagnetic field (e.g., used for electromagnetic tracking), etc. As yetanother example, the NFC interface 1732 and the network interface 1734may be part of the same communication system.

Examples of Configuring Placement and Movement of Light Patterns of aHalo

The illuminated halo described herein can be customized by a user of awearable system, a developer, or another entity. For example, the userof the wearable system can set preferences associated with the placementand movement of light patterns of the halo. The preferences may beassociated with the contextual information described with reference toFIGS. 16A and 16B. As an example, the user can set the color of a haloassociated with a system notification (such as a low battery status) toblue while setting the color associated with a game object to red. Theuser can set his or her preferences via the wearable device alone or incombination with the totem. Based on the user's preferences, the totemcan automatically present the illuminated halo when a user interactswith the wearable system. In some situations, the user can turn off theilluminated halo by actuating the totem (such as the trigger 1212 shownin FIG. 12A) or via poses. For example, the outward-facing imagingsystem 464 can capture a hand gesture of the user associated withturning off the halo. The wearable system can recognize the hand gesturefrom the images acquired by the outward-facing imaging system using acomputer vision algorithm and accordingly instruct the totem to turn offthe illuminated halo.

The placement and movement light patterns of the halo can also beconfigured using an application programming interface (API). FIGS.18A-18D illustrate an example programming interface for configuringlight placement or movement patterns of a halo. The API can beinteracted with by a user or developer to set the light patterns of thehalo of the totem. FIG. 18A shows an API that includes a visualization1802 of a totem. The visualization 1802 of the totem includes an imageof a halo 1810 displayed on a visualized touchpad 1804. In this example,the length of the halo 1810 is set to be 0.25, which can be about aquarter of the top circumference of the visualize touchpad 1804. The endpoints of the halo 1810 are at the 12 o'clock position and the 3 o'clockposition.

The programming interface 1800 can include an application bar 1830. Theapplication bar 1830 can include the types of applications for which thelight patterns of the halo are configured. For example, in FIG. 18A, theuser of the programming interface 1800 is configuring the light patternsassociated with a game application. The user can also configure thelight patterns associated with a web application, a system application,a scene (such as, e.g., the locations of the objects in theenvironment), a user's characteristics (such as, e.g., associating thepattern of a halo based on whether the user's right-handed orleft-handed), and so on.

The programming interface 1800 can also include an arc configuration tab1820. The arc configuration tab 1820 can include various options forconfiguring the shape (e.g., arcs, rings, etc.), size (e.g., angular arclength), color, visual effects, or orientation (e.g., N, S, E, or W) ofthe halo. Visual effects can include fade-in, fade-out, flash, blink,clockwise or counter-clockwise rotation, etc. For example, theprogramming interface 1800 can associate the halo 1810 (or a portion ofthe halo) with a fade-in or fade-out effect. The programming interface1800 can also change the arc length, for example, from ¼ of thecircumference to ¾ of the circumference of the halo. It can also changethe rotation of the halo 1800 (e.g., from clockwise tocounter-clockwise) or change the location of the halo 1800 (e.g., fromthe top right corner to the bottom right corner of the touch surface).

As an example of configuring in light placement or movement patterns ofthe halo using the arc configuration tab 1820, FIG. 18B shows an examplehalo 1812 when the length of the arc is set to be −0.25. In thisexample, the length of the arc in halo 1812 is about ¾ of thecircumference of the touch surface. The end points of the halo 1812 areat the 9 o'clock position and the 6 o'clock position.

The programming interface 1800 can also include a color selection tool1830 shown in FIG. 18C. The color selection tool 1830 can include asearch bar 1832 in which a user of the programming interface can inputthe color of the halo. The color of the halo may be input in the formatof Hypertext Markup Language (HTML) color code. The color selection tool1830 can also include a color adjustment panel 1834 in which the usercan adjust the saturation, brightness, and hue of a color. The colorselection tool 1830 can also include an RGB tool 1836 which can selectthe color based on the relative ratio among RGB and the black/whitecolor. The user can use a pointing device (e.g., a mouse) to selectcolor, saturation, brightness, hue, etc. from the panels of the colorselection tool.

In the example shown in FIG. 18C, the user has set the color of the halo1814 to blue. The user can also change the color of the halo 1814 usingthe color selection tool 1830 or adjust the saturation, brightness, orhue of the selected color using the color adjustment panel 1834. In someembodiments, the color of the halo may be preset, e.g., based on a themeor style sheet. The wearable system may preset the color based on asystem default color or based on contextual information. The user of theprogramming interface 1800 can change or adjust the preset color usingthe color selection tool 1830.

FIG. 18D illustrates another portion of the programming interface 1800,which includes a visualization tab 1842, a source code tab 1844, and apattern adjustment tool 1850.

The pattern adjustment tool 1850 can be used configure the halo. Forexample, the user of the programming tool can adjust the slidersassociated with the arc length, rotation, and color to provide variousvisual effect of a halo. The user can also adjust the position androtation of the halo by setting an x, y, z coordinate values halo (asillustrated in the transformation tab 1852). The user can also adjustthe lighting and movement effects associated with the totem, such as thefade-in/fade-out effect.

The visualization tab 1842 may be used to visualize one or more effectsof the halo created by the pattern adjustment tool 1850, the arcconfiguration tab 1820, or the color selection tool 1830. As an example,the visualization tab 1842 may be associated with the visualization 1802of the totem. When a user of the programming tool 1800 selects thevisualization tab 1842, the programming tool 1800 may show thevisualization 1802 of the totem so that the user can directly perceivethe effect of the updates to the halo.

The source code tab 1844 includes source codes associated with theplacement and movement light patterns of the totem. For example, whenthe user of the programming tool 1800 configures the halo using the arcconfiguration tab 1820, the color selection tool 1830, or the patternadjustment tool 1850, the source codes may be automatically updatedbased on the user's selection. The source codes (or the executableversion) can be communicated to the totem using a wired connection(e.g., via the connection interface 1222) or a wireless connection(e.g., via the network interface 1734). Additionally or alternatively,the source codes (or the executables) can be stored in the totem or thehead-mounted device (such as, e.g., in the local processing & datamodule 260), which may be used to generate analog signals to control thelight patterns of the LEDs 1330.

In some embodiments, a halo may be composed of multiple arcs (seeexample halo pattern 1614 in FIG. 16A). The programming tool 1800 canallow a user to configure each arc using the techniques describedherein. The pattern adjustment tool 1850 shown in FIG. 18D can be usedalone or in combination with the arc configuration tab 1820 and thecolor selection tool 1830 to configure the halo.

Although the examples in FIGS. 18A-18D are described with reference toconfiguring the light placement or movement patterns of the halo,similar techniques can also be used to configure haptic, audio, visual,or other effects.

Examples of Totem Calibration with a Halo

User input devices (such as the totem described herein) may requirecalibration to minimize errors and to improve user interactionexperiences. For example, the IMUs of the totem may need to becalibrated to accurately detect and respond to the user's movement ofthe totem. The calibration is normally done at the time of manufacture(e.g., when the totem or the IMUs are manufactured). Post-manufacturecalibration may be difficult because a user may not have a controlledand precise calibration system as the manufacturer of the totem has.However, it may be beneficial for a user to perform post-manufacturecalibration because the degree of the user's motion may vary. Forexample, one user may move his or her arm to a greater degree than otherusers when the user sways his or her totem. In addition, it may beadvantageous to provide a customized user experience so that the totemcan have varying degrees of responsiveness tailored to different users.Some users may prefer a more sensitive totem (which may be responsive toa slight movement) while other users may prefer a less sensitive totem(which may not be configured to perform a user interface operation whenthere is a slight movement).

The wearable system and the totem described herein can advantageouslysupport post-manufacture calibration without requiring the user to usethe calibration system of the manufacturer. The totem can display ahalo. When the user moves the totem 1200, the outward-facing imagingsystem 464 of the wearable system can capture an image of the totem andthe illuminated halo. The shape of the halo as captured by theoutward-facing imaging system 464 may be different based on the positionand orientation of the totem. The wearable system can use objectorganizers 708 to identify the shape of the halo and accordinglydetermine the position and orientation of the totem. To calibrate an IMUof a totem, for example, the position and orientation of the totem asdetermined based on the images acquired by the outward-facing imagingsystem 464 can be compared with the position and orientation captured bythe IMU. The wearable system can correct the IMU of the totem based onthe comparison.

FIG. 19A illustrates an example of totem calibration using anilluminated halo. The example in FIG. 19A is described with reference tocalibrating a totem 1900 but this is for illustration and is notintended to be limiting. The totem 1900 can have 3 degrees of freedom inthis example. The 3DOF totem 1900 can include a circular touch surface1962 and a circular light guide 1964 surrounding the touch surface 1962.The light guide 1964 can display a circular halo. The 3DOF totem 1900can be an example of the totem 1200 described herein. Accordingly, thetouch surface 1962 can be an example of the touch surface 1262 and thelight guide 1264 can be an example of the light guide 1264.

In this example FIG. 19A, various poses (e.g., poses 1910, 1920, 1930,and 1940) of the totem are shown. A pose of a totem can include anorientation and a position of the totem. For example, the pose 1920indicates that the user has pointed the totem downward. The pose 1930shows that the totem is pointed to the right and the pose 1940 showsthat the totem faces the top-right position. The pose 1910 may be afiducial position which is associated with the natural resting positionof the totem.

During a calibration process for an IMU of the totem, a wearable systemcan use its outward-facing imaging system 464 to capture an image of theuser's environment. The image of the user's environment can include animage of the totem. The object recognizer 708 of the wearable system canapply a computer vision algorithm to identify the totem in the capturedimage. The object recognizer 708 can also apply a computer visionalgorithm to identify the shape of the halo in the image. The wearablesystem can determine the position and orientation of the totem based onthe shape of the halo alone or in combination with the image of thetotem body. For example, the wearable system can identify a shapeassociated the fiducial position and calculate the changes to the shapeassociated with the fiducial position. Based on the changes to the shapeassociated with fiducial position, the wearable system can determine thechanges of the totem's position and orientation based on the fiducialposition. For example, a halo having the shape of a circle can bechanged the shape of an ellipse if the normal to the circular shapepoints in a direction that is not directly toward the imaging camera.The wearable system can accordingly determine the current position andorientation of the totem based on the changes to the fiducial position.

As an example, the illuminated halo shown by the light guide 1964 canhave a circular shape when the totem 1900 is at the fiducial position(as shown by the pose 1910). In addition, the outward-facing imagingsystem 464 may not perceive the body of the totem 1900 when the totem isat the fiducial position. With reference to the pose 1940, the wearablesystem can identify that a portion of the totem's body in a diagonalposition based on the images acquired by the outward-facing imagingsystem 464. The wearable system can also identify that the illuminatedhalo at the pose 1940 of the totem has appears to have an elliptical,rather than a circular shape. Based on the observations of the totembody and the halo (among other possible information), the wearablesystem can determine that the totem is tilted to the right and is facingup. As another example, the shapes of the halos at poses 1920 and 1930can both be ellipses. However, the major axis of the halo at the pose1930 is in a vertical direction (and the minor axis at the pose 1930 isin a horizontal direction) while the major axis of the halo at the pose1920 is in a horizontal direction (and the major axis at the pose 1920is in a vertical direction). Therefore, the wearable system maydetermine that the totem is positioned horizontally (e.g., facing leftor right) at the pose 1930 and determine that the totem is positionedvertically (e.g., facing up or down) at the pose 1920.

The fiducial position may be defined by the wearable system. In thisexample, the fiducial position is defined to be associated with the pose1910. In other examples, the fiducial position may be associated withother poses (such as poses 1920, 1930, 1940, etc.) of the totem.Although in this example, the halo appears to be a circle at thefiducial position, the totem can illuminate other light patterns fortotem calibration, which the wearable system can detect and analyze. Forexample, the totem can illuminate a square shape, a triangle (e.g., with3 arcs surrounding the touch surface of the totem), a pattern ofoverlapping line segments (having for example cross-shaped arcs or ‘x’shaped arcs), and so on. In certain embodiments, the light patterns mayalso include arcs with multiple colors (which may be useful forcalibrating the totem with more degrees-of-freedom, such as, e.g., a6DOF totem). For example, where the light pattern includes cross-shapedarcs, the totem can illuminate the light patterns as illustrated in FIG.25B. The outward-facing imaging system 464 and the object recognizers702 can use the similar techniques to determine the position andorientation of the totem regardless of the light patterns of a halo.

The wearable system can also receive data related to the totem'sposition and orientation from the IMU. For example, the wearable systemcan calculate the totem's orientation based on the data acquired from agyroscope in the totem. The position and orientation data acquired bythe IMU can be compared with those calculated based on the images of thetotem. The wearable system can calibrate the IMUs based on thiscomparison. For example, the wearable system may determine that thetotem 1900 is at pose 1940 based on the data acquired by the IMU.However, the position of the totem 1900 as observed by the wearablesystem may actually be at the pose 1930. Accordingly, the wearablesystem can associate the position and orientation observed by the IMUwith the pose 1930 (as observed by the outward-facing imaging system464) to calibrate the IMUs. In some embodiments, as an addition oralternative to calibrating the IMUs, the wearable system may store acalibration conversion (e.g., a lookup table) that corrects IMU readingsso that they represent the calibrated (rather than raw) reading.

To facilitate the calibration process, a halo may be used to guide auser to move the totem to a certain pose (e.g., position ororientation). The wearable system can accordingly acquire the positionor orientation of the totem using the outward-facing imaging system andthe IMU. FIG. 19B illustrates an example of totem calibration where thelight patterns are used as a guide for user movement. For example, thetotem can show the patterns 1972, 1974, 1976, 1978. These patterns cancorrespond to MUX 2, 0, 1, 4 shown in FIG. 14B. For example, LED 3, LED4, and LED 5 in MUX 1 may light up to show the pattern 1976.

During a calibration process, the totem presents the pattern 1972 toindicate to the user to move the totem rightward and presents thepattern 1974 to indicate to the user to move the totem leftward. Whenthe totem presents the pattern 1976, the user can move the totem upward,but when the totem presents the pattern 1978, the user can move thetotem downward. Once the totem (or the IMU) has been calibrated, thetotem can present the pattern 1980 which shows a solid blue colorforming a circular shape around the totem's touch surface. In additionto or in alternative to the pattern 1980, the totem can also employother indications to show that the totem (or the IMU) has beencalibrated. For example, the totem can provide a vibration (e.g., viathe actuator 1760) or a sound (e.g., via the speaker 240). As anotherexample, the totem can provide other visual indications, such as afade-in or fade-out effect, a portion of a circular halo, or othervisual schemes. In some embodiments, the totem (e.g., the processor1770) can instruct the LEDs to stop illuminations when the calibrationis completed.

Although FIG. 19A is described with reference to calibrating an IMU in a3DOF totem 1900, similar techniques can also be used to calibrate a 6DOFtotem or other types of user input device.

Process 1982 in FIG. 19C illustrates another example of the calibrationprocess. The patterns 1984 a, 1984 b, 1984 c, and 1984 d indicate thecurrent position/movement that the totem is currently calibrating. Forexample, the pattern 1984 a (which shows an arc on the top of a circulartouchpad) can indicate that the totem is currently calibrating itselfassociated with an upward movement while the pattern 1984 b (which showsan arc on the bottom of a circular touchpad) can indicate that the totemis currently calibrating itself associated with an downward movement. Asanother example, the pattern 1984 c (which shows an arc on the right ofa circular touchpad) can indicate that the totem is currently beingcalibrated with respect to a rightward movement and the pattern 1984 d(which shows an arc on the left of a circular touchpad) can indicatethat the totem is currently being calibrated with respect to a leftwardmovement.

In some embodiments, the patterns 1984 a, 1984 b, 1984 c, and 1984 d maybe triggered due to user interactions. For example, if the user movesthe totem rightward, the totem can display the halo 1984 d andaccordingly calibrate the totem associated with this rightward movement.Additionally or alternatively, as described with reference to FIG. 19B,the patterns 1984 a, 1984 b, 1984 c, and 1984 d can automatically be litup to guide the user to move the totem to a certain pose.

Once the totem has calibrated on all four directions (left, right, up,and down), the totem can present a confirmation pulse. The confirmationpulse may include displaying the patterns 1986 a and 1986 b inalternate. For example, the totem can change halo from the pattern 1986a to the pattern 1986 b (and verse vice) every 0.1 second.

Although FIGS. 19A and 19B are described with reference to calibratingan IMU, similar techniques can also be used to calibrate otherenvironmental sensors 1710.

In addition to or in alternative to guide a user through a totemcalibration process, similar techniques can be used to calibrate auser's interactions with a specific object. For example, when the userstarts to play a game, the light placement or movement patterns of ahalo may be used to guide the user through a tutorial process. As anexample, when totem shows the pattern 1984 c, the user can move thetotem leftward and the display can accordingly show a leftward movementof an avatar. The user can also associate the totem's movement with aparticular user interface operation. For example, the user can perceivehis or her avatar is in a running action after the avatar is created.The user can move the totem forward (away from the user) to set theforward movement of the totem to be associated with the running action.

In some embodiments, the light placement or movement patterns of thehalo may be different based on the type of calibration. For example, ahalo may be in a blue color when the IMU is calibrated while may be agreen color when the user is configuring the movement of the totems fora game.

Example Processes of Totem Calibration

FIG. 19D illustrates an example process 1990 of totem calibration usinglight patterns associated with a halo. The example process 1990 can beperformed by the wearable system (such as, e.g., the local processing &data module 260 or remote processing module 270), alone or incombination with the totem.

At block 1992 a of the example process 1990 in FIG. 19D, the wearablesystem can receive an image of a user's environment. The image may beacquired by the outward-facing imaging system 464 shown in FIG. 4 .

At block 1992 b, the wearable system can analyze the image to identify atotem (or a halo) in the image. The totem can include a body and a lightguide. The light guide may be configured to display a halo. The wearablesystem can use the object recognizers 708 described with reference toFIG. 7 to identify the totem, the body of the totem, a halo, or thelight guide. The object recognizers 708 may use computer visionalgorithms to perform such identification.

At block 1992 c, the wearable system can determine a first position ofthe totem. The first position may be a fiducial position. The fiducialposition may be pre-defined by the wearable system, such as, e.g., thenatural resting position of the totem. The fiducial position may also beidentified or changed by the user. For example, a user may define thefiducial position to be the position associated with the pose 1910. Thewearable system can determine a first light pattern of the haloassociated with the fiducial position. The pattern associated with thefiducial position may be previously stored by the wearable system. Insome embodiments, once the wearable system identifies the fiducialposition, the wearable system may prompt a notification to the userrequesting the user to hold the totem at the fiducial position and thewearable system can accordingly capture an image of the totem at thefiducial position.

At block 1992 d, the wearable system can identify a second pattern ofthe halo based at least partly on an analysis of the image. The wearablesystem can analyze the image and identify the pattern using the objectrecognizers 708.

At block 1992 e, the wearable system can determine a first change to thetotem's position and orientation with the respect to the first position.For example, the wearable system analyze the change to the halo'spattern based on the pattern associated with the fiducial position andthe pattern acquired by the outward-facing imaging system when the totemis at the updated position. In addition to the image analysis, thewearable system can also identify the second pattern of the halo orcalculate the first change to the totem's position and orientation basedon other information, such as the user interactions or data from one ormore environmental sensors which are not being calibrated. For example,the user can provide an indication to show the change to the totem'sposition and orientation. The indication can be via poses or the userinput device 466, such as, e.g., pointing a finger to the frontindicating that the totem has been moved forward from its originalposition.

At block 1994 a, the wearable system can also receive movement dataassociated with the totem from the environmental sensor that needs to becalibrated. The wearable system can accordingly calculate a secondchange to the totem's position and orientation based on the datareceived from the environmental sensor at block 1994 b.

The wearable system can calculate the differences between the firstchange and the second change. At block 1992 f, if the wearable systemdetermines that the discrepancy between the first change and the secondchange passes a threshold condition, the wearable system can calibratethe environment sensor at block 1992 g. Otherwise, the wearable systemcan determine that the environmental sensor does not need to becalibrated and provide an indication as shown in block 1992 h. Theindication may include a certain placement and movement of the halopattern such as, e.g., the confirmation pulse shown in FIG. 19C, orhaptic feedback. Additionally, the indication may be presented by thedisplay 220, the speaker, etc.

In some situations, the wearable system can gather data associated withmultiple positions and orientations of the totem, the wearable systemcan determine whether the environmental sensor needs to be calibrated orhow the environmental sensor will be calibrated based on the gathereddata. For example, the wearable system may determine that the IMU doesnot need to be calibrated when the totem is moving left and right butmay need to be calibrated with respect to forward/backward movement.However, in order to calibrate for the forward/backward movement, thewearable system may also need to adjust the IMU for the detectingleft/right movement. Accordingly, the wearable system may calculate theamount of adjustments needed for the forward/backward movement as wellas the left/right movement even though the IMU may have already beenable to detect left/right movement with reasonable accuracy.

FIG. 19E illustrates another example process 1996 of totem calibrationusing light patterns associated with a halo. The example process 1996can be performed by the totem or the wearable system described herein.

At block 1998 a, the totem can provide a first indication of a totem's apose. For example, the totem's light guide may display a halo with anarc located on the left side of the light guide. This arc can provide anindication to the user to move the totem to the left. In someembodiments, the indication can also be provided by the wearabledisplay, or via haptic or audio feedback.

At block 1998 b, the totem can acquire movement data associated with thepose. For example, as the user moves the totem leftward, the totem's IMUcan detect the user's movement and communicate the movement data to aprocessor.

At block 1998 c, the totem can be calibrated based on the movement data.For example, the totem can associate the movement detected by the IMUwith a leftward motion.

At block 1998 d, the totem can determine whether it has been calibrated.For example, if the IMU of the totem needs to be calibrated, the totemcan determine whether the totem has been calibrated for 6DOF. Once thetotem has been calibrated, at block 1998 f, the totem can present asecond indication informing that the totem is calibrated. For example,the totem can show a flashing green halo on the light guide to indicatethat the totem is calibrated.

If the calibration process is unsuccessful, the process 1996 may returnto the block 1998 a to repeat the calibration.

Examples of Device Pairing with a Halo

Two devices may be required to establish a wireless communication linkbefore they are paired up (e.g., via Bluetooth). When devices pair up,they can share information such as, e.g., addresses, names, and profilesof the users of the devices or the devices themselves or share theinformation stored in their respective memories. Once the two devicesare paired, they can share a common secret key, which can allow them tobond again for future information sharing.

The device pairing can be tedious and sometimes require the users toenter unique information about their devices. For example, a user maymanually input information of his or her device after his or her device(or the other user's device) completes an initial search for nearbydevices. Although the devices can use NFC protocols to share informationbetween the two devices for the pairing process, NFC components areoften relatively large and costly.

To reduce or minimize the user interaction and simplify the pairingprocess between a totem and a wearable device, the totem can present alight pattern which can be analyzed and used for pairing. The lightpattern can include a message used for pairing between the totem and thewearable device. The light pattern may be presented by a haloilluminated on the totem's LEDs. The message may include an indicationto initiate a pairing process. For example, the light pattern can encodea trigger message which can cause the wearable device to search fornearby devices or to broadcast its device information to nearby devices.As another example, the light pattern can encode device information ofthe totem. The device information may be encrypted. Device informationcan include a device identifier (ID), operating system information(e.g., operating system version), firmware information, type ofcommunication channel (e.g., Bluetooth, wireless channels) used forpairing, a digital key shared between the totem and the wearabledisplay, etc. In certain implementations, the device ID may be providedor generated by the manufacturers, distributors, developers, or anysuitable entity. Examples of device identifier may include Androididentifier (ID), iPhone's Unique Identifier (UDID), iPhone'sIdentifierForAdvertising (IFA or IDFA), cookie ID, login ID, InternetProtocol (IP) address, media access control (MAC) address, a hash of anyof the above, a combination of any of the above, or the like. In somecases, the device identifier may be derived based on one or morehardware and/or software parameters of a device identified by the deviceidentifier. For example, a device identifier may be derived from the IPaddress, operating system version, and locale setting of the device. Insome embodiments, a device identifier may be used to identify the sourceor origin or a transaction, request, or network event. For example, adevice identifier may include a user identifier, an account identifier,and the like. Additionally or alternatively, a device ID may beassociated with an entity (e.g., user) using a device.

As an example, the totem may include a ring of LEDs on its perimeter(e.g., 12 LEDs disposed circumferentially around the touchpad). These 12LEDs can be used to represent certain device information when they lightup in a specific temporal sequence, spatial sequence, or a combinationthereof. Each LED may be assigned to a bit and the ring of LEDs canrepresent, in this example, a 12 bit word. One or more of the LEDs canbe display in sequence to show the necessary information for pairing.For example, the LED lights can be illuminated in a sequence to show adevice identifier (ID) of the totem. Assuming the device ID is 2, andthe binary representation of the number 2 is 10, the totem may light theLED 0 (shown in FIG. 14B) to represent the binary number 2 while otherLEDs (e.g., LED 1-11) do not illuminate.

In certain implementations, the colors of one or more LED lights canalso represent information used for pairing. For example, a red colormay represent a certain manufacturer or model of the totem while a bluecolor of the LED lights may represent another manufacturer or model. Thetotem can also show a combination of manufacturer and model of the totemwith the LED lights. For example, the totem can show manufacturer A andmodel M by illuminating LED 1 in blue while illuminating LED 3 in pink.

Furthermore, where each LED is assigned to a bit, the color of the LEDscan expand the amount of information that the LEDs can represent. Forexample, rather than one LED being assigned to one bit, an LED with acertain color may be assigned to one unit of information. Accordingly,if an LED light can illuminate 9 colors, the LED can be used to show thenumbers 0-9 based on the illuminated colors. For example, when the LEDis not illuminated, the wearable device can interpret it to be 0, whilethe color purple may represent the number 1, the color green mayrepresent number 2, etc. As another examples, the LEDs can alsorepresent letters in an alphabet (e.g., red being A, white being B,etc.).

The wearable device may be associated with a display and a camera (suchas, e.g., the outward-facing imaging system 464). The camera can be usedto capture the light sequence and then demodulate it to extract theusable information for automatic pairing. For example, the imagescaptured by the camera can be analyzed using various computer visionalgorithms described herein to extract the light patterns of the halo.The light patterns may be translated to a number, alphabet, word, etc.,which can represent the device information of the totem. The wearabledevice can then use this device information for pairing with the totem.

Two totems may also be paired. For example, two friends may want to pairtheir totems to share their game controls while they are playing a videogame using their respective wearable devices. In this example, theoutward-facing imaging system of the first person's wearable device canobtain the light sequence on the second friend's totem (and vice versa).In first person's wearable device can extract device information of thesecond friend's totem and pass that information to the first friend'stotem for pairing. In some implementations, a totem may also have acamera. Thus, a totem can capture the images of light patterns ofanother totem and achieve pairing of the two totems.

As another example, a wearable device (or an HMD) may be paired with twototems. The outward-facing imaging system can acquire images of thelight patterns for one totem. Once the wearable device extracts thedevice information of one totem from its halo, the wearable device canpass the device information (of the wearable device or the other totem)to the other totem for pairing.

In addition to or in alternative to device pairing, similar techniquesfor communicating information by the totem can also be used tocommunicate other types of information. For example, the totem maycommunicate a user's profile information, or status of the totem to thewearable device using the light patterns illuminated by the totem.

Advantageously, in some embodiments, the light placement or movementpatterns of the totem can be used to indicate the progress of thewireless pairing between a totem and a wearable device or between twototems. FIGS. 20A and 20B illustrate examples of indicating a wireless(e.g., Bluetooth or WiFi) pairing process with a halo. The lightpatterns illustrated in these two figure can be used to show theprogress of a device pairing process via wireless communications (e.g.,where a totem or device broadcasts it device information to otherdevices wirelessly) or through the light patterns described withreference to FIG. 14B.

FIG. 20A shows a totem 2020 which can have a touchpad 2022 and a displayregion 2026. The totem 2020 may be an example of the totem 1200 and thetouchpad 2022 may be an embodiment of the touchpad 1260. The displayregion 2026 may include a ring of LEDs and a light guide diffusing thelight emitted by the LEDs. In some embodiments, the user can actuate thedisplay region 2026.

The display region 2026 can display a halo 2024. A portion of the halomay be in the dark blue color. As shown in the process 2012, the totemcan play a spin animation of the dark blue portion clockwise. Once thetotem is paired, the totem can present a dark blue halo with a pulseanimation. An example of the animations is described with reference toFIG. 20B. The totem may start the spin animation by displaying thepattern 2062. While the wireless pairing is in progress, the totem canpresent the patterns 2064, 2066, and 2068 sequentially as illustrated byarrows 1, 2, and 3. The totem can play the pattern 2062 again after thepattern 2068 if the pairing is still ongoing. If the pairing hascompleted successfully, the totem can present the pattern 2070 which canprovide a visual indication to the user.

Although these examples are described with reference to pairing a totem,similar techniques can also be used for pairing other user input devices466 or devices with display or illumination components. For example, thetechniques can be applied to pair two or more of: a smartwatch, asmartphone, a wearable device, a totem, a tablet, a computer, atelevision, a camera, a peripheral device, an appliance, or otherdevices that have image recognition and processing capacities. As anexample of pairing between a smartphone and the totem, the smartphonecan use its camera to capture and analyze the illumination patterns onthe totem. As another example, a wearable device may be paired with asmartwatch where the wearable device can capture and analyze a patterndisplayed by the smartwatch (e.g., on a display screen of thesmartwatch). The techniques described herein can further be applied tovarious types of wireless pairings such as Bluetooth pairing, WiFipairing (e.g., Wi-Fi Direct™), or other types of wireless pairings. Asan example of Bluetooth pairing, when the totem is in discoverable mode,the totem can display a light pattern indicating that it is in thediscoverable mode. The wearable device can capture this light patternand complete the pairing when the wearable device is also in thediscoverable mode. When the light pattern encodes a triggering message,the wearable device can automatically enter the discoverable mode upondetection of the light pattern. As an example of wireless pairing, thetotem can illuminate a light pattern indicating that a connection can bemade to the totem. The wearable device can receive an invitation toconnect to the totem upon detection of the light pattern. In addition tothese examples of wireless pairings, the pairing techniques can also beapplied to types of pairings other than the wireless pairing. Forexample, the techniques can also be used to share files among a group ofuser input devices at home via Ethernet connection.

Example Processes of Device Pairing with a Halo

FIG. 20C illustrates an example process 2080 of device pairing with ahalo. The pairing may be between a wearable device and another device(e.g., a totem, a smartphone, a television, a smartwatch, etc.). Theexample process 2080 can be performed by a wearable device having an HMDand an outward-facing imaging system 464. The process 2080 can be usedto assist pairing a totem with other devices, HMDs, etc.

At block 2082, the wearable device can receive a request from a totem toestablish a pairing. For example, the totem can broadcast a signal tonearby devices for establishing a communication. The wearable device canreceive the broadcast signal and identify the totem which hascommunicated the signal. As another example, the request may be atrigger message encoded in the light pattern of the totem.

At block 2084, the wearable device can receive an image of a halopresented by the totem. The image may be captured by the outward-facingimaging system 464. As described with reference to FIGS. 20A and 20B,the totem can use the halo to communicate device information needed forestablishing the pairing via the light placement or movement patterns ofthe halo. For example, the halo may encode device information to abinary form.

At block 2086, the wearable device can analyze the image of the halo toextract device information. For example, the wearable device can useobject recognizers 708 n to identify the halo in the captured image.Where the halo encodes the device information in a binary form, thewearable device can convert the binary representation back to a decimalrepresentation or an alphabetical representation of the deviceinformation.

At block 2088, based on the extracted device information, the wearabledevice can pair the totem with the wearable device. For example, thewearable device may add the device ID of the totem to a whitelist ofdevices with which the wearable device can share its information.

Optionally at block 2089, the wearable device can receive an indicationfrom the totem showing that the pairing is successful. The indicationmay be an electro-magnetic signal. Additionally or alternatively, thetotem can display a halo with a certain placement and movementindicating that the pairing is complete. For example, the totem candisplay a pulse animation of a circular blue halo. The outward-facingimaging system 464 can capture this halo in an image. Once the wearabledevice recognizes the light placement or movement pattern of the halo,the wearable device can determine that the pairing has been completed.

In some embodiments, the totem also provides a visual indication thatthe pairing is in progress. For example, the totem may present a halowith a rotating arc during the pairing. The rotating arc may bepresented after the totem has provided the device information to thewearable device.

Although the examples are described herein with reference to a wearabledevice, in some embodiments, similar techniques can also be applied withanother computing device. For example, the example process 2080 can beused to pair a cellphone configured to provide AR/VR/MR features with atotem.

FIG. 20D illustrates another example process 2090 of device pairing witha halo. The pairing may be between a totem and a computing device. Thecomputing device may be a wearable device having a wearable display andan outward-facing imaging system 464, or other computing devices (e.g.,a television, a smartwatch, a smartphone, tablet, another totem, etc.).The example process 2090 can be performed by the totem 1200.

At block 2092, the totem can initiate a pairing process with a computingdevice. For example, the totem can search for nearby devices that can bepaired with the totem. The totem can also broadcast a signal andidentify available devices for pairing based on the response to thesignal received from other devices. The user can actuate the totem orthe computing device to cause the totem to initiate the pairing process.For example, the user can press the home button 1256 for an extendedperiod of time to cause the totem to enter a pairing process. In certainimplementations, the totem can present a light pattern which indicatesthat the totem is now in the pairing mode. The light pattern may becaptured by the computing device and can cause the computing device toenter into the pairing process with the totem.

At block 2094, the totem can illuminate a first light pattern encodinginformation for the pairing process 2094. The encoded information mayinclude device information of the totem. The encoded information mayalso include a key for the computing device to connect with the totem.For example, the computing device can extract the key based on the lightpattern extracted from the image and communicate the key back to thetotem, via a wireless channel to achieve pairing. In some embodiments,the block 2092 may be optional. The computing device can automaticallyinitiate the pairing process upon detection of the first light patternin the block 2094.

At block 2096, the totem can receive a response from the computingdevice. The response may be the device information of the computingdevice. In some situations, the response may include a request for moreinformation from the totem. The computing device can also provide otherinformation in the response, such as, network communication information(e.g., a network port, an encryption standard, or a communicationprotocol) that the totem can use to communicate with the computingdevice. In some situations, the response can also include an indicationthat the computing device has added the totem to a list of accepteddevices.

Optionally at block 2098, once the pairing process is completed, thetotem can illuminate a second light pattern indicating that the pairingis successful. The second light pattern may be the pattern 2070 shown inFIG. 20B. In some embodiments, the totem can stop illuminating toindicate that the pairing has completed. Although the example process2090 has been described above as being performed by a totem, it is to beunderstood that the example process 2090 can be performed by anotherdevice (e.g., a smartphone, a smartwatch, a computer, etc.).

Examples of Using Light Patterns of a Halo to Provide a Cue for anObject's Status

The placement and movement of a halo's light patterns can be used to asa cue for indicating the status of an object. For example, certain lightplacement or movement patterns can be used to indicate whether a userhas turned the totem on or off, whether the totem is in sleep mode, orwhether the battery charging status of the totem. Non-limiting,illustrative examples are described below.

FIG. 21A illustrates an example of indicating the status of the totem.As shown in the process 2110), when the power button is turned from offto on, the totem can present a portion of the halo and progressivelyfill the rest of the portions until the totem is fully turned on. Thetotem can also rotate the halo clockwise for one cycle during the poweron process.

When the power button is switched from on to off, the totem can present(as illustrated in the process 2120) a full halo and rotate the halocounter-clockwise for one cycle. The totem can also progressively hidemore portions of the halo until the totem is completely powered off(when the totem will no longer present a halo).

The halo color palette can be selected to be visually pleasing to users.Different color palettes can be used for different system functions,e.g., a startup color palette when the system is powered up (see, e.g.,the process 2110 in FIG. 21A) and a shutdown color palette when thesystem is powered down (e.g., the process 2120 in FIG. 21A). Forexample, the startup color palette can represent colors typically seenduring sunrise, and the shutdown color palette can represent colorstypically seen during sunset. Other color palettes can be used for othersystem functions (e.g., charging or battery status or a sleep-idle mode,shown in FIGS. 21C, 21D). Color palettes can include colors, grayscales, illumination levels, etc. Illumination patterns can includeillumination information that is communicated to a light sourcecontroller (which might be a controller disposed on the touch board 1394c) used to control the brightness of the totem light sources (e.g., LEDs1383 a, 1394 a). The illumination information can include a colorpalette, timings (or rates of change) of when, for how long, and at whatbrightness level each of the LEDs (including individual colorsproducible by the LED) are illuminated, etc. Some LEDs may includeadditional or different control functionality, e.g., RGBA, whichincludes RGB plus an Alpha channel that indicates how opaque each pixelis and permits the totem to perform alpha compositing (allowing theappearance of partial or full transparency). The illuminationinformation can include such additional or different controlfunctionality, depending on the choice of light source used in thetotem.

There can be an enormous variety of color palettes that can be displayedby RGB LEDs (e.g., where, commonly, each color can be set to 256 values,from 0 to 255). Color palettes can be selected for one or more of thehalos described herein on the basis of any of a variety of differentfactors, e.g., user characteristics, contextual information, deviceusage or status, system functionality, etc.

The halo illumination pattern can include gradients in colors, shading,hues, etc. For example, due to the limited number of light sources inthe totem (e.g., 12 RGB LEDs), varying the brightness of the coloredlight source can be used to provide a desired gradient (in color orintensity). A gradient can provide the visual illusion of intermediateendings or fadings out of an illumination pattern. The gradient can beprovided by adjusting the brightness of the light sources dynamically(for time-varying illumination patterns). As an example, for a totemwith 12 LEDs positioned at hours of the clock, an illumination patternstarting at 4 o'clock that extends 45 degrees toward 6 o'clock, with agradient at the 6 o'clock end of the arc, could have LED brightnessvalues set as: LEDs 1-3, brightness 0.0, LED 4, brightness 1.0, LED 5brightness 1.0, LED 6 brightness 0.5, LEDs 7-12, brightness 0.0.

Other shutdown processes can involve user interaction with the totem, toconfirm that the user wants to perform a shutdown operation (e.g., toshutdown, suspend, or restart the wearable device 200). For example, thedisplay 220 of the wearable system 200 can display to the user ashutdown user interface when shutdown is selected by the user. In oneexample, the shutdown user interface displays a circular ring(suggestive of the circular light guide 1264 on the totem 1200) and atextual indication that the user is to move his or her thumb on thetotem's touchpad 1260 to confirm the shutdown operation. For example,the user interface may display “Complete Circle to Power Off” and thecircular ring may include an animation indicating the user is to movetheir thumb (say) clockwise around the touchpad 1260. If the userpresses the home button 1256, the shutdown operation may be halted andthe shutdown user interface may fade out.

If the user wants to complete the shutdown operation, the user canfollow the displayed directions and move his or her thumb in a circular,clockwise path around the touchpad 1260. As the user does so, the lightguide 1264 can illuminate to create an arc that follows the thumb'smotion (and optionally, the display 220 can display an animated graphicshowing progress around the circular ring). If the user's fingercompletes a full circle (confirming the shutdown operation), a haptic“bump” can be performed by the totem 1200 to provide the user withtactile feedback that the shutdown operation has been started(optionally, the system 200 can output an audio sound). The display 220of the wearable system 200 can display a shutdown screen to the user andbegin the shutdown operation. The light guide 1264 of the totem 1200 candisplay a shutdown illumination pattern (e.g., as shown in FIG. 21B).

FIG. 21B illustrates an example of light placement or movement patternsof the halo during a power on and off process. During the power onprocess, the halo pattern can move from the pattern 2122 to the pattern2124 and then to the pattern 2126 as shown by the arrows 1 and 2. Duringthis process, the arc of the halo (which may be illuminated in a color,e.g. a yellow color) can get progressively longer in length and brighterin color.

On the other hand, during the power off process, the halo patterns canmove from patterns 2126 to 2124 then to 2122 as shown by the arrows 3and 4. In this process, the shape of the halo gradually can reduce froma full ring to a small arc.

Additionally or alternatively, the halo can be configured to show thecharging status of the battery 1720 of the totem, where the amount ofbattery may correspond to the size of the portion of the halo beingdisplayed. FIG. 21C illustrates an example of placement and movement oflight patterns which show a battery charging status. Process 2132illustrates an animation sequence of the halo when the battery 1720 ischarging. For example, the halo may be divided into four quadrants. Eachquadrant may be associated with one or more LED lights. The totem canprogressively fill all four quadrants by illuminating the LEDs in eachquadrant. In certain implementations, rather than filling 1 quadrant ata time, the totem can gradually illuminate the LEDs to track thepercentage of the battery remaining. For example, when there is 30% ofthe battery remaining, the totem can light up 30% of the LEDs.

If the battery is below 15% (for example) when the charging begins, thetotem can present a halo with a first color (e.g., an orange color) inone of the four quadrants. Once the battery has reached 25% (forexample), the totem can change the color of the halo from the firstcolor to a second color (e.g., from orange to green). When the batteryhas reached 50% (for example), the totem can illuminated the portion ofthe halo (e.g., in the second color) associated with the next quadrantin the clockwise direction. The totem can continue fading in otherquadrants as the totem is being charged. When the battery is fullycharged, the totem can present a halo in the second color with all fourquadrants illuminated (as shown in the process 2134).

While the battery is being consumed, the shape of the halo may initiallybe a full circle. The totem can gradually fade the halo out in thecounter-clockwise direction as illustrated in the process 2136 as theamount of power decreases. In some embodiments, when there is only 15%(for example) of the battery remaining, the totem may turn the remainingportion of the halo to a third color (e.g., a red color) and play apulsing animation until no power is remaining (where the totem can playa fade-out animation of the halo).

The halo can also be used to show whether the totem has entered into asleep mode. FIG. 21D illustrates an example halo pattern when the totemhas entered into the sleep mode. As indicated in the process 2142, whenthe totem has become idle because it has not detected user interactionswithin a threshold period of time, the totem can present a halo in theform of a trail (e.g., a white trail). The totem can also repeat thetrail in a clockwise cycle.

Example Processes of Indicating a Status of a Totem with a Halo

FIG. 21E illustrates an example process of indicating a status of atotem based on light placement or movement patterns. The process 2150 inFIG. 21E can be performed by one or more components of the wearablesystem described herein. For example, the process 2150 can be performedby a totem or a wearable device.

At block 2152, the wearable system can determine a status of a totem.The totem can include a group of light sources (e.g., LEDs) configuredto display a halo. The halo may be used to indicate the status of thetotem. The status may include the battery status of the totem, such as,e.g., whether the totem is powered on/off, whether the totem is in asleep mode, or the progress of battery charging or consumption.

At block 2154, the wearable system can access a first light pattern ofthe halo. The first light pattern may specify a placement or movementpattern for the halo. The first light pattern can be associated with thecurrent status of the totem. For example, when the battery is currentlylow, the light placement or movement pattern of the halo may be a redarc around the 6 o'clock position.

At block 2156, the wearable system can instruct the group of LEDS todisplay the halo in accordance with the first light pattern. Continuewith the previous example, the wearable system can generate and send aninstruction to the LEDs located near the 6 o'clock positions (such as,e.g., the LED 9, the LED 10, and the LED 11 in the MUX 3 shown in FIG.14B) to illuminate a red light.

The wearable system can continuously monitor the status of the totem.Optionally at block 2158, the wearable system can determine whetherthere is an update to the status. For example, while the battery of thetotem may previously be less than 15%, the user may have plugged thetotem into a power source for charging. Accordingly, the amount ofbattery may gradually increase.

At the optional block 2160, in response to the update, the wearablesystem can instruct the group of LEDs to display the halo in accordancewith a light pattern. The second light pattern is associated with theupdate. For example, once the totem has entered into the charging mode,the color of the halo may be changed from red to green.

Although the examples herein are described with reference to indicatingthe status of the totem, similar techniques can also be used to indicatethe status of other components of the totem such as the wearable displayor the battery pack.

Examples of Using Light Patterns as a Cue for User Interactions

In addition to or in alternative to providing information on devicestatus, the halo can also be used as a cue to indicate a current userinteraction or to guide a user's interaction. FIGS. 22A and 22Billustrate example light placement or movement patterns that are used ascues for user interactions. FIG. 22A displays halo patterns 2210 and2220. The halo patterns may be display by a light guide around a touchsurface of a totem. The user can actuate the touch surface, for example,using his or her thumb 2230. The halo pattern 2210 can include an arc2222 (which may be illuminated in a purple pink color) on the left sideof the light guide. The length and brightness of the arc 2222 mayreflect the distance between the user's finger and the portion of thelight guide displaying the arc 2222.

As an example of indicating a current user's interactions with thetotem, when the user's thumb is at position 2212, the halo pattern 2210may be shown. The portion of the arc 2222 that is closest to the thumb2230 may be the brightest. The brightness of the other portions of thearc may gradually reduce as they are farther away from the thumb 2230and eventually transitioned into black (e.g., where the correspondingLEDs are turned off). In this example, the user can actuate the touchsurface with a swipe gesture. For example, the user can swipe his or herthumb leftward. As the user moves his or her thumb 2230 leftward towardthe arc 2222 (illustrated by the arrow 2214), the length of the arc 222may increase from less than half of a circle to more than half of thecircle). The brightness of the arc 2222 can also increase due to thereduced distance between the thumb 2230 and the arc 2222.

As another example, in FIG. 22B, the halo associated with the pattern2250 can include three arcs 2252, 2254, 2256. The arc 2252 isillustrated in a first pattern corresponding to a first color (which,e.g., may be a red), the arc 2254 is illustrated in a second patterncorresponding to a second color (which, e.g., may be blue), and the arc2256 is illustrated in a third pattern corresponding to a third color(which, e.g., may be yellow). With reference to FIG. 14B, the arc 2252may be illuminated by LED 1, LED 2, and LED 3; the arc 2254 may beilluminated by LED 5, LED 6, and LED 7; and the arc 2256 may beilluminated by LED 9, LED 10, and LED 11.

The pattern 2250 may be associated with a 3-way d-pad, with each arcbeing associated with a user interface operation. For example, the usercan tap the region near the arc 2252 to move a virtual object leftwardand tap the region near the arc 2254 to move the virtual objectrightward. The user can also tap the region near the arc 2256 to selector release the virtual object. When the user's thumb 2230 is near anarc, the length of the arc may decrease while the brightness of the arcmay increase. For example, when the user's thumb 2230 is near the arc2254, the totem may turn off the LED 5 and LED 7 (shown in FIG. 14B) toreduce the size of the arc 2254. This change to the light placement ormovement pattern of the halo can provide an indication that the user hasinitiated (or is about to initiate) a leftward movement of the virtualobject. When the user's thumb moves away from the arc 2254, the totemmay re-illuminate the LED 5 and LED 7 to increase the size of the arc2254.

A totem can simulate functions of various user input devices, such as,e.g. a d-pad, a touch device, a 6DOF controller, etc. The lightplacement or movement patterns of the halo can be used to indicate thetype of user input device that the totem is currently being used as orto indicate available user interface interactions. For example, while auser is browsing a webpage using the 3D display, the totem can be usedas a touch device which supports a left/right swipe gesture. To providean indication to the user that the swipe gesture is available, the totemcan display the patterns shown in FIG. 22A.

In certain implementations, the wearable system can automatically selecta suitable user input device configuration based on the contextualinformation. For example, the user may want to use a 3-way d-pad to movevirtual objects in and out of the FOV and to select a virtual object.Accordingly, when the wearable device detects that the user isinteracting with the virtual objects in his or her environment, thewearable device can instruct the totem to enable 3-way d-pad and presentthe halo pattern 2250 to inform the user that the 3-way d-pad isavailable.

FIG. 22C illustrates another example of using light patterns associatedwith a halo to provide an indication of an available user interfaceoperation. The totem in FIG. 22C can be configured to serve as astar-shaped d-pad. To inform the user that the totem is currently beingused as a star-shaped d-pad, the totem may light up the LED 2, LED 4,LED 6, LED 8, LED 10, and LED 0 in FIG. 14B. As an example, when a userselects a text box on the webpage, the totem may present the pattern2260 to show that the user can use the star-shaped d-pad for text input.When the user actuates a region near an LED, the LED may display abrighter color to indicate that the user is actuating the region. Forexample, when the user actuates the region near the LED 0 (asillustrated in the pattern 2270), the totem may instruct the LED 0 toemit a brighter light.

In some embodiments, the totem may use the totem to inform the user thatthe user is performing a wrong user interaction or provide an indicationof a correct/recommended user interaction. FIG. 23 illustrates anexample of using a light pattern as an alert to indicate an incorrect orimproper user operation. The user's thumb 2230 may initially be at theposition 2312 on a touch surface. The touch surface may be surrounded bya light guide which can display the halo 2222. When user's thumb is atthe position 2312, the light guide can display the halo pattern 2310.The user may need to move an object leftward by swiping to the left(e.g., by moving his or her thumb 2230 closer to the halo 2222. However,the user actually swipes rightward (as indicated by the arrow 2232),which is the incorrect direction to perform the desired operation ofmoving the object leftward. As a result the user's thumb is now at theposition 2314. To remind the user that he or she needs to swipe to theleft to properly move the object, the light guide may show the halopattern 2320 where the halo 2222 appears to be longer in length andbrighter in color. This can draw the user's attention to make sure thathe or she moves his or her thumb 2230 in the correct direction.

The patterns of the halo can vary depending on the type of userinteraction. FIGS. 24A and 24B illustrate example patterns for a swipegesture and a touch gesture respectively. The touchpad in FIGS. 24A and24B can include a light guide which displays a halo. The touchpad canalso include a touch surface which is surrounded by the light guide. Thetouch surface can receive user input such as a swipe gesture or a touchgesture.

The halo shown in FIGS. 24A and 24B can have 3 arcs 2422 (illuminated ina first color, e.g., red), 2424 (illuminated in a second color e.g.,blue), and 2426 (illuminated in a third color, e.g., yellow). Withreference to FIG. 14B, the arc 2422 may correspond to the LED 1, LED 2,and LED 3; the arc 2424 may correspond to the LED 5, LED 6, and LED 7;and the arc 2426 may correspond to the LED 9, LED 10, and LED 11.

The user's thumb 2230 may initially be at the position 2412 of the touchsurface. The light guide can accordingly display the pattern 2402 forthe halo. As indicated by the downward arrow 2450, the user can swipedownward toward the arc 2426. For example, the user can move his or herthumb 2230 from the position 2412 to the position 2414 (as shown byarrow 1). Upon detecting of the user's movement, the totem can dim thedisplay of the arc 2422 and 2424 as shown in the pattern 2404. The usercan further move his or her thumb downward as indicated by the arrow 2.In the pattern 2406, when the user's thumb reaches the position 2416,the totem can increase the brightness and length of the arc 2426. Forexample, the brightness of the LED 9, LED 10, and the LED 11 mayincrease. The totem can also illuminate the LED 0 and LED 8 (in thethird color) to increase the length of the arc 2426. The totem canfurther illuminate the LED 7 in the third color (rather than the secondcolor as shown in arc 2424) and illuminate the LED 1 in the third color(rather than the first color as shown in arc 2422). As user moves his orher thumb 2230 even further downward as shown by the arrow 3, all LEDs(except LED 4 which is on the opposite side of LED 10) can illuminate alight in the third color to show the expansion of the arc 2426 in thethird color. On the other hand, when the user's thumb 2230 moves awayfrom the arc 2426, the totem may display the patterns in accordance withthe arrows 4, 5, and 6. Under this sequence, the size of the arc 2426can gradually shrink and the other arcs 2422 and 2424 can gradually fadein.

The user can also actuate the touchpad in FIG. 24A with a touch gesture.FIG. 24B illustrates the halo patterns when the user actuates thetouchpad with the touch gesture. The totem may initially display 3 arcs2422, 2424, and 2426 as shown in pattern 2462. When the user touches aregion close to the yellow arc 2426, the length of the arc 2426 mayincrease. As illustrated in the pattern 2464, when the user actuates thetotem with a touch gesture at position 2418, the LED 0 and 8 may beilluminated (in the third color) to show the increased length. However,the arc 2426 will not expand to cover up other arcs (e.g., arcs 2422 and2424) as compared to the pattern 2408 (which can correspond to a swipegesture), even though the user's thumb is at the same position 2418.

Example Process of Providing a Cue for User Interactions on a Totem

FIG. 24C illustrates an example process of providing a cue for userinteractions on a totem. The example process 2480 may be performed bycomponents of the wearable systems, such as, e.g., a totem or a wearabledevice.

At block 2482, the wearable system can identify available user interfaceoperations based on contextual information. For example, the wearablesystem can determine what types of user interactions are available andmap these types of user interactions to the interactable regions of thetotem. As an example, a browser can allow a user to scroll up/down,enlarge/shrink the size of a webpage, select an item on the browser, andmove a cursor around within the browser. The wearable system can map thecursor movement and item selection to a first interactable region of thetotem, where the user can swipe a touch surface to move the cursor andclick the touch surface to select the item. The wearable system can alsomap the scrolling and resizing operations to a second interactableregion. The second interactable region may surround the firstinteractable region. The second interactable region may simulate thefunctions of a d-pad, where the user can tap the up/down keys to scrollthe webpage and tap the left/right keys to resize the webpage.

At block 2484, the wearable system can determine a placement or movementpattern of a halo associated with the available interface operations.Continuing with example above, the wearable system can display a halowith 4 arcs (as shown in FIG. 25B) to indicate that the totem has asecond interactable region that the user can use as a d-pad.

At block 2486, the wearable system can generate and transit instructionsto LEDs (shown in FIG. 14B) of a totem to display a halo in accordancewith the light pattern. For example, where halo has 4 arcs, the wearablesystem can instruct the LED 1 to illuminate a blue color, LED 4 toilluminate a yellow color, LED 7 to illuminate a red color, and LED 10to illuminate a green color.

At block 2488, the wearable system can receive a user input on thetotem. For example, the totem can detect a swipe gesture or a touchgesture in the first interactable region or detect a tap gesture in thesecond interactable region. In some embodiments, the user input may alsoinclude actuating other components of the totem, such as the trigger1212 or the home button 1256 (shown in FIG. 12 ) alone or in combinationof user's poses (e.g., head pose or other body poses).

Optionally at block 2492, the wearable system can determine whether theuser input is the correct input. The wearable system can make thedetermination based on the contextual information of the object withwhich the user is interacting. The wearable system can determine whetherthe user input belongs to the types of user inputs supported by theobject. For example, a video recording application can allow user inputsvia poses while a web browser can support a combination of swipe andtouch gestures. Accordingly, when a user is browsing a webpage, thewearable system may not perform a user interface operation based on achange of the user's foot pose. As another example, one webpage mayallow a user to view the web content but may not allow a user to inputhis or her comments. Accordingly, if the user tries to type a phrase,the wearable system may determine that the user's input is not thecorrect input because it is not supported by the webpage.

If the user input is not the correct input, optionally at block 2494,the wearable system can provide an indication that the user's input isincorrect. The indication may be provided in the form of visual, audio,haptic, or other feedback. For example, the totem can emphasize thecorrect input by providing a brighter color near the interactable regionassociated with the correct input. As another example, the totem mayprovide a vibration via a haptic actuator in the totem's body. As yetanother example, the wearable system can provide an alert message via anHMD or via the speaker.

If the user input is the correct input, at block 2496, the wearablesystem can update the placement and movement of the light pattern toreflect the user input. For example, if a user taps the left side of thetotem's touch surface, the totem may light up the LED near the user'stap. In some embodiments, once the totem receives the user's input, thetotem can update the replacement and movement of the light patternassociated with the halo to reflect the user input, in addition to or inalternative to blocks 2492 and 2494.

Examples Interactions with Virtual Objects Using a Totem

The totem can include multiple interactable regions where eachinteractable region can be mapped to one or more types of userinteractions or to a type of user input device. As described withreference to FIGS. 12A and 13A, in some embodiment, the totem have afirst interactable region comprising a touch surface which can supporttouch events and gestures, such as e.g., a swipe gesture, and a secondinteractable region comprising a light guide. The light guide canilluminate a halo at 4 points around the touchpad or with 4 arcs eachhaving a different color. This illumination pattern can show that thelight guide is used as a d-pad. For example, the user can tap or presson the point of illumination to move a virtual object. The same usergesture can cause different user interface operations being performeddepending whether the gesture is applied to the light guide or the touchsurface (e.g., in the main center area of the touchpad). For example,the user may move a virtual object when user taps the light guide, butwhen the user taps the touch surface, the user may select the virtualobject.

FIG. 25A illustrates an example interactive use of the light guide. Thetotem 2502 in FIG. 25A may be an example of the totem 1200 describedherein. The totem 2502 can include a touchpad 2506 which can include atouch surface and a light guide, which can substantially surround thetouch surface. The light guide can diffuse lights (e.g., emitted by LEDsunderneath the light guide) to show a halo, which may include one ormore arcs of a halo. In the example in FIG. 25A, the halo can include 4arcs 2504 a, 2504 b, 2504 c, and 2504 d illuminated at 4 regions aroundthe touch surface. The light guide can be interactable so as to receiveuser input (e.g., the user might depress a portion of the light guide).For example, the light guide can simulate the functions of a d-pad. Thelight guide can include an upward key 2508 a, a right key 2508 c, a downkey 2508 d, and a left key 2508 b. The totem can provide visualindication on the locations of these keys. For example, the arc 2504 acan correspond to the upward key 2508 a; the arc 2504 b can correspondto the left key 2508 b; the arc 2504 c can correspond to the right key2508 c, and the arc 2504 d can correspond to the downward key 2508 d.

A user can actuate the regions near the respective keys to perform userinterface functions related to these keys. For example, a user canactuate the region 2518 a to actuate the upward key 2508 a; actuate theregion 2518 b to actuate the left key 2508 b; actuate the region 2518 cto actuate the right key 2508 c; and actuate the region 2518 d toactuate the downward key 2508 d.

As an example of actuating the light guide, the user can move thevirtual objects in and out of his or her FOV by actuating the lightguide. Scene 2500 in FIG. 25A schematically illustrates an example ofinteracting with virtual objects in the FOV 2550 and the FOR 2502. Asdescribed with reference to FIG. 4 , an FOR can comprise a portion ofthe environment around the user that is capable of being perceived bythe user via the wearable system. The FOR 2502 can include a group ofvirtual objects (e.g. 2510, 2520, 2530, 2542, and 2544) which can beperceived by the user via a display 220. Within the FOR 2502, theportion of the world that a user perceives at a given time can bereferred to as the FOV 2550 (e.g., the FOV 2550 may encompass theportion of the FOR that the user is currently looking toward). In thescene 2500, the FOV 2550 is schematically illustrated by the dashed line2552. The user of the display can perceive multiple objects in the FOV2550, such as the object 2542, the object 2544, and a portion of theobject 2530. The FOV 2550 may correspond to the solid angle that isperceivable by the user when looking through the display such as, e.g.,the stacked waveguide assembly 480 (FIG. 4 ) or the planar waveguide 600(FIG. 6 ). As the user's pose changes (e.g., a head pose or an eyepose), the FOV 2550 will correspondingly change, and the objects withinthe FOV 2550 may also change. For example, the virtual object 2510 isinitially outside the user's FOV 2550. If the user looks toward thevirtual object 2510, the virtual object 2510 may move into the user'sFOV 2550, and the virtual object 2530 may move outside the user's FOV2550.

The user can actuate light guide to move virtual objects in and out ofthe FOV 2550 without changing his or her body poses. For example, theuser can actuate the key 2508 b to move the virtual objects in the FOV2550 leftward. Accordingly, the virtual object 2530 may be movedentirely into the FOV 2550. As another example, the user can actuate thekey 2508 c to move virtual objects rightward and as a result, thevirtual object 2510 may be moved into the FOV 2550 and the virtualobject 2530 may be moved outside of the FOV.

Additionally or alternatively, the wearable system can assign a focusindicator to a virtual object in the FOV 2550 based on the user'sdirection of gaze or the location of the virtual object. For example,the wearable system can assign the focus indicator to a virtual objectthat intersects with the user's direction of gaze or to a virtual objectthat is closest to the center of the FOV. The focus indicator cancomprise an aura (e.g., around the virtual object), a color, a perceivedsize or depth change (e.g., causing the virtual object to appear closerand/or larger when selected), or other visual effects which draw theuser's attention. The focus indicator can also include audible ortactile effects such as vibrations, ring tones, beeps, etc. For example,the wearable system may initially assign the focus indicator to thevirtual object 2544. When the user taps at the region 2518 c, the usercan actuate the key 2508 c. Accordingly, the wearable system can movethe virtual objects rightward to transport the focus indicator from theobject 2544 to the object 2542, and the wearable system can move theobject 2510 into the user's FOV 2550 while moving the object 2530 out ofthe user's FOV.

FIG. 25B illustrates an example interactive use of a totem with twointeractable regions. As described with reference to FIG. 25A, the totem2502 can include the touch surface 2506 and the light guide. The lightguide can simulate the functions of a d-pad as described with referenceto FIG. 25A and can display a halo having 4 arcs 2514 a, 2514 b, 2514 c,and 2514 d, with each arc having a different color.

The user can actuate the light guide or the touch surface 2506 toactuate the totem. For example, in a browser window, the user canactuate the touch surface 2506 to move a cursor. A left swipe on thetouch surface 2506 can be mapped to the back function while a forwardswipe on the touch surface 2506 can be mapped to the forward functionfor the browser page. A tap (illustrated by the position 2518) at the 6o'clock position (illuminated with the arc 2514 d) can cause the browserto scroll down and a tap at the 12 o'clock position (illuminated withthe arc 2514 a) can cause the browser to scroll up. Holding at the 6o'clock position or the 12 o'clock position can quickly scroll the page.

The totem can provide visual feedback when the user actuates the totem.For example, when the user taps at the 6 o'clock position near the arc2514 d, the totem can increase the brightness of the arc 2514 d. Hapticfeedback can be used to distinguish whether the user has actuated thetouch surface 2506 or the light guide. For example, when a user taps onthe light guide, the actuator under the touchpad may be activated andprovide a vibration.

FIG. 25C illustrates an example process for interacting with a totem.The process 2570 may be performed by the wearable system (such as thetotem or the wearable device) described herein.

At block 2572, the wearable system can receive a user input on a totem.The user input may include a hand gesture, such as a swipe, a tap, atouch, a press, etc. The totem can comprise a plurality of interactableregions. For example, the totem can include a touchpad which can includea touch surface and a light guide, both of which may be interactable.

At block 2574, the wearable system can determine an interactable regionassociated with the user input from the plurality of interactableregions. For example, the totem's touch sensor 1350 can determinewhether the user has actuated the light guide or the touch surface. Thetouch sensor 1350 can also detect the user's gestures used for actuatingthe totem. For example, the touch sensor can detect whether the useractuates the totem using a swipe gesture or a tap gesture.

At block 2576, the wearable system can access a light pattern associatedwith the user input detected in the interactable region. As describedwith reference to FIGS. 24A and 24B, the light patterns may be differentbased on the type of user input (e.g., a touch gesture v. a tapgesture). The light patterns may also be different depending on whichinteractable region is actuated. For example, if the user actuates thetouch surface, the light patterns may be those illustrated in FIG. 24Abut if the user actuates the light guide, the light patterns may includethose shown in FIG. 24B.

At block 2578, the wearable system can instruct the totem's lightsources (e.g., LEDs) to present a halo comprising the light pattern. Forexample, the wearable system can provide instructions on which LEDSshould light up, the brightness of the LEDs, the colors of the LEDs, themovement patterns associated with the light emitted by the LEDs, etc.

Examples Interactions with Physical Objects Using a Totem

FIGS. 26A and 26B illustrate examples of interacting with physicalobjects using a totem. FIG. 26A illustrates an example of a physicalenvironment 2600 which may be a living room of a user's home. Theenvironment 2600 has physical objects such as, e.g., a television (TV)5110, a physical remote control 5120 (sometimes simply referred to as aremote), a TV stand 5130, and a window 5140. The user can perceive thephysical objects and interact with the physical objects. For example,the user may watch the TV 5110 and control the TV using the remote 5120.The user can control the remote 5120 to turn the TV 5110 on/off orchange the channel or volume of the TV 5110.

The user can also interact with the TV 5110 using the totem 2602. Thetotem 2602 can be an embodiment of the totem 1200, which can includemultiple interactable regions. The totem 2602 may be paired with the TV5110 or the remote 5120 to simulate the functions of the remote 5120.The functions of the remote 5120 may be mapped to one or moreinteractable regions associated with the touchpad, a trigger, or a homebutton of the totem. The touchpad can include a touch surface 2634 and alight guide 2630, where the touch surface 2634 and the light guide 2630may be mapped to different types of user interactions. For example, thetouch surface 2634 can be used to switch channels via a swipe gesture.The light guide 2630 can be used as a d-pad for adjusting volumes orfast forward/backward via a tap gesture.

In FIG. 26B, the light guide can show a light pattern 2622 for a halohaving four arcs: the arc 2632 a (which may be in a first color, e.g.,blue), the arc 2632 b (which may be in a second color, e.g., yellow),the arc 2632 c (which may be in a third color, e.g., red), and the arc2632 d (which may be in a fourth color, e.g., green). The arcs arepositioned to form an x-shape. As illustrated by arrow 1, when the user2602 touches a region near the arc 2632 c on the light guide, the usercan fast forward the program being played on the TV 5110. The totem canshow the pattern 2624 to indicate that that the user has tapped near thearc 2632 c. The arc 2632 c appears to be brighter and longer while otherarcs 2632 a, 2632 b, and 2632 d are hidden in the pattern 2624.

The user can also change to the next channel by swiping to the right (asillustrated by arrow 2. The totem can illuminate a light pattern 2626 toshow that the user has actuated the totem by a swipe gesture rather thana tap gesture. Even though the user's 2602 finger is at roughly the sameposition in the patterns 2626 and 2624, the light patterns illuminatedin these two situations are different because the gestures used toactuate the totem are different. For example, although the brightnessand length of the arc 2632 c have increased, other arcs 2632 a, 2632 b,and 2632 d are not hidden in the pattern 2626. Rather, the totem merelyreduces the brightness of the other arcs.

Although the examples described with reference to FIGS. 26A and 26B aredescribed without a wearable device, in some embodiments, the user cancontrol a physical object using a totem while wearing the wearabledevice. For example, the user can perceive physical objects through thewearable device and perceive virtual objects projected by the wearabledevice. The user can interact with the physical objects as well as thevirtual objects using the totem.

Examples 6DOF User Experiences with a Totem

FIG. 27 illustrates an example of moving a virtual object with amulti-DOF (e.g., 3DOF or 6DOF) totem. The user can change the positionof the totem by moving forward/backward (surging), up/down (swaying), orleft/right (heaving). The user can also change the orientation of thetotem by titling side to side (rolling), titling forward and backward(pitching), or turning left and right (yawing). By changing the positionand orientation of the totem, the user can perform user variousinterface operations (such as, e.g., by moving on rotating) on virtualobjects.

A user may hold and move a virtual object by selecting the virtualobject using the totem 2602 and move the virtual object by physicallymoving the totem 2602. For example, the totem 2602 may initially be at afirst position 5100 a. The user 5300 may select a target virtual object5400 located at a first position 5100 b by actuating the totem 504(e.g., by actuating a touch sensitive pad on the totem). The targetvirtual object 5400 can be any type of virtual object that can bedisplayed and moved by the wearable system. For example, the virtualobject may be an avatar, a user interface element (e.g., a virtualdisplay), or any type of graphical element displayed by the wearablesystem. The user 5300 can move the target virtual object from the firstposition 5100 b to a second position 5200 b along a trajectory 5500 b bymoving the totem 2602 along a trajectory 5500 a.

As another example, instead of moving the virtual object 5400 from 5100b to 5200 b, the user 5300 may want to pull the virtual object 5400closer to himself. Accordingly, the user can move the totem to closer tohimself to bring the virtual object 5400 closer.

As yet another example, the totem 2602 may be at the position 5200 a.The user can rotate the totem for 180 degrees clockwise, and the virtualobject 5400 can accordingly be rotated for 180 degrees at the position5200 b.

The totem can emit certain light patterns indicating the type of userinteraction. For example, the ring of LEDs may light a blue color when auser is moving the totem while light a green color when the user isrotating the totem. As another example, the totem light movement patternmay correspond to the rotation of the totem. For example, the lightpattern may include an arc. As the user rotates the totem, the arc canalso rotate in the direction of the user's rotation.

Examples of Decorative and Informative Visual Feedback on a Totem

Some wearable devices can have a limited FOV. Advantageously, in someembodiments, the totem can provide information associated with objectsoutside of the FOV using the light patterns illuminated by the totem.

FIGS. 28A and 28B illustrate examples of providing information ofobjects via placement and movement of light patterns. FIG. 28Aillustrates a person's FOV 2820 and FOR 2810. The FOR 2810 can contain agroup of objects (e.g., objects 2812, 2822, 6050 b, 6050 c) which can beperceived by the user wearing the wearable system. The FOV 2820 cancontain multiple objects (e.g., objects 6050, 2822). The FOV can dependon the size or optical characteristics of the wearable system, forexample, clear aperture size of the transparent window or lens of thehead mounted display through which light passes from the real world infront of the user to the user's eyes. In some embodiments, as the user's6100 pose changes (e.g., head pose, body pose, and/or eye pose), the FOV2820 can correspondingly change, and the objects within the FOV 2820 mayalso change. As described herein, the wearable system may includesensors such as cameras that monitor or image objects in the FOR 2810 aswell as objects in the FOV 2820. In such embodiments, the wearablesystem may alert (for example, via the light patterns on the totem) theuser of unnoticed objects or events occurring in the user's FOV 2820 oroccurring outside the user's FOV but within the FOR 2810. In someembodiments, the wearable system can also distinguish between what theuser 6100 is or is not directing attention to.

The objects in the FOV or the FOR may be virtual or physical objects.The virtual objects may include, for example, operating system objectssuch as e.g., a terminal for inputting commands, a file manager foraccessing files or directories, an icon, a menu, an application foraudio or video streaming, a notification from an operating system, andso on. The virtual objects may also include objects in an applicationsuch as e.g., avatars, virtual objects in games, graphics or images,etc. Some virtual objects can be both an operating system object and anobject in an application. The wearable system can add virtual elementsto the existing physical objects viewed through the transparent opticsof the head mounted display, thereby permitting user interaction withthe physical objects. For example, the wearable system may add a virtualmenu associated with a medical monitor in the room, where the virtualmenu may give the user the option to turn on or adjust medical imagingequipment or dosing controls using the wearable system. Accordingly,wearable system may present additional virtual image content to thewearer in addition to the object in the environment of the user.

The totem can provide information of objects that are outside of the FOVvia the placement and movement of light patterns. The placement andmovement of light patterns can be calculated and determined by variouscomponents of the wearable system, such as, e.g., the totem or thewearable device. The wearable system can use contextual informationassociated with the user or the objects to determine the light patterns.For example, the wearable system can employ factors such as, e.g., thelocation of the object (including the proximity of the object relativeto the user or the user's FOV), the urgency of the object, the type ofthe object (such as, e.g., whether an object is interactable, what typeof interactions are supported by the object, whether the object isphysical or virtual), the property of the object (such as e.g.competitor's avatar v. friend's avatar), the volume of information (suchas, e.g., the number of notifications), the user's preference, etc. Asshown in the scene 2834 in FIG. 28B, because the object 2840 is furtheraway from the user 6100 than it is in the scene 2832, the halo 2844 onthe totem 2502 may appear to be smaller (e.g., have a shorter angularextent) than the halo 2842 (which has a larger angular extent). Asanother example, a halo may have a larger and brighter appearancebecause the object associated with the halo is more urgent or closer tothe user's FOV.

The wearable system may assign a color to an aura based on thecharacteristics of the associated object. For example, the wearablesystem may assign a red color to the aura 6051 b because the object 6050b is associated with the red color. Similarly, the wearable system mayassign a blue color to the object 6051 a because it is an operatingsystem object and the wearable system assigns the blue color to alloperating system objects.

The wearable system may assign a color to a halo based on thecharacteristics of the associated object. For example, the wearablesystem may assign a green color to the halos 2842 and 2844 because theobject 2840 is associated with the green color. Similarly, the wearablesystem may assign a red color to the object 6050 b because it is anoperating system object and the wearable system assigns the red color toall operating system objects. As a result, the halo associated withoperating system objects (including the object 6050 b) may be in a red.

The appearance of the halo may change over time based on a change of theobject associated with the halo. For example, a halo may grow thicker asthe object associated with the halo receives more messages. As anotherexample, the halo 2844 may get bigger (or brighter) as the object 2840moves closer to the user or may grow smaller (or dimmer) as the object2840 moves away from the user. The position of the halo may also changeas the objects moves. For example, in FIG. 28A, the wearable system mayinitially show a halo associated with the object 6050 b on the left sideof the totem's light guide because the object 6050 b is to the user's6100 left. But when the object 6050 b moves to the right side of theuser 6100, the wearable system can update the position of the halo tothe right side of the totem's light guide.

The appearance of the visual aura may also change based on a change ofthe user's pose. For example, with reference to FIG. 28A, as the user6100 turns leftwards, the object 6050 b may become closer to the user'sFOV while the object 6050 c. As a result, the halo associated with theobject 6050 b may become brighter (or larger).

In some situations, the FOR 2810 and the FOV 2820 can include invisibleobjects that a user cannot directly see. For example, a treasure huntgame may include hidden treasures (e.g. objects 2812, 2822 illustratedin dashed lines) in the user's FOR 2810. The treasure may not bedirectly visible to the user because it may be buried in a game object.However, the user can use a virtual tool in the game (such as e.g., avirtual shovel) to reveal the hidden object. As another example, auser's physical car keys may be underneath a stack of papers, and theuser may not be able to perceive the physical car keys in the user'sFOV. The halo can be used to provide an indication of the location ofthe hidden car keys. For example, the size and location of an arc of thehalo can indicate the relative distance and orientation between the carkeys and the totem. The location of the car keys may be determined basedon the map of the user's environment (e.g., based on images acquired bythe outward-facing imaging system at previous times) or may bedetermined based on a wireless communication between the totem (or othercomponents of the wearable system) with the car keys where the keys areequipped with radio or wireless communication capacities.

As yet another example, objects (e.g., object 6050 a) may be outside theuser's visual FOR but may nonetheless potentially be perceived by asensor (e.g., an electromagnetic sensor, a radio frequency sensor, asensor associated with detecting wireless signals, or anotherenvironmental sensor) on the wearable device or the totem. For example,the object 6050 a may be behind a wall in a user's environment so thatthe object 6050 a is not visually perceivable by the user. However, thewearable device or the totem may include sensors that can communicatewith the object 6050 a.

The totem can provide information of invisible objects via the lightpatterns. The placement and movement of the light patterns can be basedon contextual information associated with the invisible objects or theuser. For example, the placement and movement of the light patterns onthe light guide can act as a compass directing the user to look or movein a certain direction. As another example, the light patterns mayprovide a map of invisible objects in the user's environment. Forexample, the totem can illuminate a 30 degree red arc at the one o'clockposition indicating the direction of a car key is at the user's frontright while illuminating a 90 degree green arc at the six o'clockposition indicating a user's cellphone is behind the user. The red arcand the green arc can also be used to indicate the proximity of the carkey and the cellphone. For example, the cellphone is closer to the userthan the car key and, as a result, the green arc is bigger than the redarc. As the user moves, the placement (or movement) of light patternscan also change. For example, as the user moves forward, the size of thegreen arc may decrease while the size of the red arc may increase.

In certain implementations, the wearable system may present a map of theuser's physical or virtual environment which can indicate a location ofthe invisible objects. For example, an HMD can present a virtual mapwhich shows an in-game object which a user can uncover. As anotherexample, the HMD can provide, as AR/MR content a focus indicator orother types of visual indications of the hidden physical car keys (or avirtual object). As the user moves around in the environment, theappearance of the focus indicator (e.g., the size and relative directionfrom the user) may change. In some situations, the HMD can present aroute to the invisible objects. The map presented by the HMD may becombined with the halo on the totem to guide the user to the invisibleobjects.

In some embodiments, the totem can present a halo which includesinformation associated with multiple objects. For example, a user 6100may have a red object 6050 b to his or her left and an invisible object2812 behind him. The totem can present a halo which has a red arc on theleft side of the light guide and a silver arc on the bottom of the lightguide, where the red arc corresponds to the red object 6050 b and thesilver arc corresponds to the invisible object 2812.

In addition to or in alternative to using light patterns to provideinformation of objects, the display can also be configured to provideinformation of objects that are outside of the FOV or are invisible. Forexample, the display can display a visual aura for a correspondingobject outside of the user's current FOV. A portion of the visual auracan be placed on the edge of the user's FOV of the display. Theplacement of the aura can be based on the contextual information of theobject.

Besides visual auras, the wearable system can also inform the user aboutobjects using a tactile or an audio effect. For example, in a mixedreality game, the wearable system may notify the user of an approachingcompetitor through vibrations on the totem. The wearable system mayprovide strong vibrations when the competitor is close to the user. Inanother example, the wearable system may use audible sounds to provideposition information of a virtual object. The wearable system may alsouse a loud sound to alarm the user of a virtual competitor which isnearby. The tactile, audio, or visual feedback can be used incombination or in alternative to inform the user about the surroundingobjects.

FIG. 28C illustrates an example process for providing informationassociated with an object using light patterns. The process 2870 can beperformed by the wearable system (such as the totem or the wearabledevice) described herein.

At block 2872, the wearable system can identify an object in the user'senvironment. The object may be a physical or virtual object. The objectmay be in the user's FOR but is outside of the user's FOV. The objectcan also be hidden from the user's view. For example, the object may bebehind another object (e.g., a wall) or may be visible only with acertain user interface interaction (e.g., when a user finishes a levelin a game).

At block 2874, the wearable system can access contextual informationassociated with the object. The contextual information may be used todetermine a movement and placement of light patterns associated with theobject as shown at block 2874. For example, the brightness of a halohaving the light pattern may be determined based on the proximity orurgency of the object. The color of the halo may also match the color ofthe object.

At block 2878, the wearable system can instruct the totem's LEDs toilluminate in accordance with the light patterns. For example, theinstructions may include which LEDS should light up, the brightness ofthe LEDs, the colors of the LEDs, the movement patterns associated withthe light emitted by the LEDs, etc.

FIG. 29A illustrates example light placement or movement patternsindicating the receipt of a notification. The patterns 2912 and 2914include an iridescent pattern. The totem can display an iridescent trailin a clockwise cycle (as shown by the process 2920) to providenotifications. The notifications may be associated with a virtualobject, such as, e.g., an email application, a game, a videoapplication, etc. The notifications may also be associated with theoperating system of the wearable device. One example notification may bean error message from the operating system of the wearable device.Additionally or alternatively, the notifications may be associated withphysical objects. For example, the iridescence patterns 2912 and 2914may indicate that the coffee machine in the user's kitchen has finishedbrewing.

The totem can repeat the iridescent trail until a threshold condition isreached. The threshold condition may be based on a duration of time, auser interaction (such as e.g., an actuation of the trigger 1212), orother contextual information.

FIG. 29B illustrates an example process for providing a notificationusing light patterns on a totem. The example process 2970 can beperformed by the wearable system (such as the totem or the wearabledevice) described herein.

At block 2972, the wearable system can determine a status of a virtualobject. For example, the status may include whether the virtual objecthas received new information (e.g., a new message), whether the virtualobject is idle, whether the virtual object has run into a problem, etc.

Optionally at block 2974, the totem can generate an alert based at leastpartly on the status. For example, the virtual object may be a messengerapplication. When the virtual object receives a new message from theuser's friend, the wearable system can generate an alert indicating thata new message has arrived.

At block 2976, the wearable system can access a light pattern associatedwith the status (or optionally the alert) of the virtual object. Theplacement or movement patterns may be determined based on the contextualinformation associated with the virtual object (or the alert), such as,e.g., the importance of the alert, the urgency of the alert, thelocation of the virtual object, the type of the virtual object, etc.

At block 2978, the wearable system can instruct the totem's LEDs toilluminate in accordance with the light pattern. For example, thewearable system can provide instructions on which LEDS should light up,the brightness of the LEDs, the colors of the LEDs, the movementpatterns associated with the light emitted by the LEDs, etc.

In addition to or in alternative to providing information on objectsthat are outside of the FOV, the light patterns can also be used toprovide information of objects that are not necessarily outside of theFOV. For example, when a user wins a level in a game, the totem maylight up a halo which has the same colors as the game. As anotherexample, with reference to FIG. 29A, the virtual application associatedwith the notification may be inside of the FOV. The user may interactwith an email application, for example, by composing a message whilereceiving a notification that a new email has arrived.

The totem can provide visual feedback of the user's current interactionto a user or another person of the user's information. For example, whena user is recording a video with their HMD, the totem can illuminate ablinking red light pattern to reinforce to the user and to indicate toothers nearby that the HMD is in a recording mode.

FIG. 30 illustrates an example light pattern that can be used to informa person in the user's environment of the user's current interaction.For example, when the user is in a telepresence session, the totem canilluminate the light pattern 3000 (e.g., a green halo with a clockwisemovement) via the light guide. The light pattern 3000 can inform thepeople in the user's environment that the user is in the telepresence.This may help other people from getting too close to the user andthereby prevent other people from interrupting the user's telepresencesession. In addition to or in alternative to light patterns, other typesof feedbacks such as, e.g., haptic, audio, video feedbacks, and so on,can also be used to indicate user's current interactions of the wearablesystem.

ADDITIONAL ASPECTS

In a 1st aspect, a system comprising: a light emitting assembly of auser input device, wherein the light emitting assembly is configured toilluminate a plurality of light patterns for providing information of anobject in an environment; a hardware processor communicatively coupledto the light emitting assembly and programmed to: identify an object inan environment of a user; access contextual information associated withthe object; determine characteristics of a light pattern to beilluminated by the light emitting assembly based at least partly on thecontextual information; and instruct the light emitting assembly toilluminate in accordance with the light pattern.

In a 2nd aspect, the system of aspect 1, wherein the object comprises atleast one of: a physical object or a virtual object.

In a 3rd aspect, the system of aspect 2, wherein the status of thevirtual object comprises at least one of: a current interaction with thevirtual object by the user, whether the virtual object has received newinformation, whether the virtual object is idle, or whether the virtualobject is in an error state.

In a 4th aspect, the system of any one of aspects 1-3, wherein thecharacteristics of the light pattern comprises at least one of: abrightness, a position, a shape, a size, or a color.

In a 5th aspect, the system of any one of aspects 1-4, wherein thecontextual information associated with the object comprises at least oneof: a location of the object, an urgency of the object, a type of theobject, a property of the object, a volume of information associatedwith the object, or a preference of the user.

In a 6th aspect, the system of any one of aspects 1-5, wherein thesystem further comprises a wearable display device, and the object isinvisible from a user's view or is outside via the wearable displaydevice, and the hardware processor is programmed to determine at leastone of a size, shape, or color of the light pattern to provide a cue tothe user for locating the object.

In a 7th aspect, the system of any one of aspects 1-6, wherein object isa component of a wearable system for presenting virtual content to auser, and the light pattern indicates a status of the component of thewearable system.

In an 8th aspect, the system of aspect 7, wherein the componentcomprises at least one of: the user input device, a wearable displaydevice, or a battery pack.

In a 9th aspect, the system of any one of aspects 1-8, wherein thestatus comprises at least one of: a battery status, a power chargingstatus, a wireless pairing status between the wearable display deviceand the user input device, a status of a calibration process of the userinput device, or a status of the wearable display device.

In a 10th aspect, the system of any one of aspects 1-9, wherein thelight pattern encodes an alert or information associated with theobject.

In an 11th aspect, the system of any one of aspects 1-10, wherein thestatus comprises at least one of: a current interaction with the objectby the user, whether the object has received new information, whetherthe object is idle, or whether the object is in an error state.

In a 12th aspect, the system of any one of aspects 1-11, wherein thecharacteristics of the light patterns is configurable by a user via anapplication programming interface.

In a 13th aspect, a light-emitting user input device comprising: a touchcomponent configured to accept a user input; a light emitting assemblyconfigured to output a plurality of light patterns, the light emittingassembly at least partially surrounding the touch component; and ahardware processor communicatively coupled to the touch component andthe light emitting assembly, and programmed to: identify a userinterface operation supported by the touch component based on contextualinformation; determine a first light pattern associated with the userinterface operation; generate and transmit instructions to the lightemitting assembly to display a halo having the first light pattern;receive a user input on the touch component; and update the halo with asecond light pattern to reflect the user input.

In a 14th aspect, the light-emitting user input device of aspect 13,wherein the contextual information comprises at least one of: anenvironment of the user, types of inputs supported by the light-emittinguser input device, information associated with objects that the handhelduser input device is configured to interact, or characteristics of awearable device associated with the handheld user input device.

In a 15th aspect, the light-emitting user input device of aspect 13 or14, wherein the light emitting assembly comprises a light guide and aring of LEDs.

In a 16th aspect, the light-emitting user input device of any one ofaspects 13-15, wherein the light emitting assembly is configured toaccept a plurality of user inputs and wherein the hardware processor isfurther programmed to display the halo based at least partly on theplurality of user inputs supported by actuating the light guide.

In a 17th aspect, the light-emitting user input device of any one ofaspects 13-16, wherein the user input comprises at least one of: aswipe, a tap, a press, or a touch gesture.

In an 18th aspect, the light-emitting user input device of any one ofaspects, 13-17, wherein the light-emitting user input device comprisesat least one of a totem, a smartwatch, or a smartphone.

In a 19th aspect, the light-emitting user input device of any one ofaspects 13-18, wherein the first light pattern provides a cue to a userthat the user interface operation is supported by the light-emittinguser input device.

In a 20th aspect, the light-emitting user input device of any one ofaspects 13-19, wherein the hardware processor is further programmed to:determine whether the received user input is improper based on thecontextual information; and wherein the second light pattern provides avisual alert that the received user input is improper in response to adetermination that the received user input is improper.

In a 21st aspect, the light-emitting user input device of any one ofaspects 13-20, wherein at least a portion of the halo appears to bebrighter or larger in the second light pattern as compared to the firstlight pattern.

In a 22nd aspect, the light-emitting user input device of any one ofaspects 13-21, wherein characteristics of the plurality of lightpatterns is configurable by a user via an application programminginterface.

In a 23rd aspect, the light-emitting user input device of aspect 22,wherein the characteristics comprise at least one of: a placement ormovement pattern, a color, a brightness, a shape, or a size of an arc.

In a 24th aspect, the light-emitting user input device of any one ofaspects 13-23, wherein the second light pattern indicates, at least oneof the following in response to the user input: a battery status, apower charging status, a wireless pairing status between the handhelduser input device and another computing device, or whether the handhelduser input device is idle.

In a 25th aspect, a method comprising: under control of a hardwareprocessor: identifying a type of user input supported by alight-emitting user input device based on contextual information,wherein the light-emitting user input device comprises a first elementfor illuminating a plurality of light patterns and a second element forreceiving user inputs; determining a first light pattern associated withthe type of user input supported; generating and transmittinginstructions to the first element to illuminate a halo having the firstlight pattern; determining a second light pattern based on a user inputon the light-emitting user input device; and updating the halo to thesecond light pattern in response to the user input.

In a 26th aspect, the method of aspect 25, wherein the first element isfurther configured to receive another user inputs, and wherein the typeof user input for determining the first light pattern is associated withthe other user inputs supported by the first element. Although aspects25 and 26 recite the first element and the second element, the wordelement may be substituted by the word portion and the word element inthese two aspects can include an individual component or a subcomponent,or a combination of components (subcomponents) of the light-emittinguser input device.

In a 27th aspect, the method of aspect 25 or 26, wherein the contextualinformation comprises at least one of: an environment of the user, typesof inputs supported by the light-emitting user input device, informationassociated with objects that the handheld user input device isconfigured to interact, or characteristics of a wearable deviceassociated with the handheld user input device.

In a 28th aspect, the method of any one of aspects 25-27, wherein thetype of user input comprises at least one of: a swipe, a tap, a press,or a touch input.

In a 29th aspect, the method of any one of aspects 25-28, wherein thefirst light pattern provides a cue to a user that the type of user inputis supported by the light-emitting user input device.

In a 30th aspect, the method of any one of aspects 25-29, furthercomprising: determining whether the user input received by thelight-emitting user input device is improper based on the contextualinformation; and wherein the second light pattern provides a visualalert that the user input is improper in response to a determinationthat the user input is improper.

In a 31st aspect, the method of any one of aspects 25-30, wherein atleast a portion of the halo appears to be brighter or larger in thesecond light pattern as compared to the first light pattern.

In a 32nd aspect, a light-emitting user input device comprising: aplurality of interactable regions configured to receive user inputs,wherein at least one interactable region of the plurality ofinteractable regions comprises is part of a light emitting assembly ofthe light-emitting user input device; and a hardware processorprogrammed to: detect a user's actuation of the light-emitting userinput device; determine an interactable region among the plurality ofinteractable regions corresponding to the user's actuation; translatethe user's actuation into a user input for performing a user interfaceoperation based at least on a type of the actuation and the interactionregion associated with the actuation; and instruct the light emittingassembly to illuminate a light pattern in response to the user input.

In a 33rd aspect, the light-emitting user input device of aspect 32,wherein the light-emitting user input device further comprise a touchsurface and the light emitting assembly comprises a light guide, andwherein the plurality of interactable regions comprises a firstinteractable region associated with the light guide and a secondinteractable region associated with the touch surface.

In a 34th aspect, the light-emitting user input device of aspect 33,wherein in response to a detection that the user actuated the firstinteractable region, the hardware processor is programmed to cause thelight emitting assembly to illuminate a first light pattern associatedwith the user input in the first interactable region, and wherein inresponse to a detection that the user actuated the second interactableregion, the hardware processor is programmed to cause the light emittingassembly to illuminate a second light pattern associated with the userinput in the second interactable region.

In a 35th aspect, the light-emitting user input device of any one ofaspects 32-34, wherein the light pattern is determined based at leastpartly on contextual information associated with the light-emitting userinput device.

In a 36th aspect, the light-emitting user input device of any one ofaspects 32-35, wherein the user's actuation comprises at least one of: aswipe, a tap, a press, or a touch gesture.

In a 37th aspect, the light-emitting user input device of any one ofaspects 32-36, wherein the light pattern comprises an arcuate regionhaving a color, an arcuate length, or a visual effect.

In a 38th aspect, the light-emitting user input device of any one ofaspects 32-37, wherein the hardware processor is further programmed tocommunicate with a wearable device causing the wearable device toperform a user interface operation based on the user input.

In a 39th aspect, a method comprising: under control of a hardwareprocessor: detecting a user's actuation of a light-emitting user inputdevice, wherein the light-emitting user input device comprises aplurality of interactable regions; determining an interactable regionamong the plurality of interactable regions corresponding to the user'sactuation; translating the user's actuation into a user input forperforming a user interface operation based at least on a type of theactuation and the interaction region associated with the actuation; andcausing a light emitting assembly of a light-emitting user input deviceto illuminate a light pattern in response to the user input.

In a 40th aspect, the method of aspect 39, wherein the plurality ofinteractable regions comprises a first interactable region supporting afirst type of user input and a second interactable region supporting asecond type of user input.

In a 41st aspect, the method of aspect 40, wherein in response to adetection that the user actuated the first interactable region, thehardware processor is programmed to cause the light emitting assembly toilluminate a first light pattern associated with the user input in thefirst interactable region, and wherein in response to a detection thatthe user actuated the second interactable region, the hardware processoris programmed to cause the light emitting assembly to illuminate asecond light pattern associated with the user input in the secondinteractable region.

In a 42nd aspect, the method of aspect 40 or 41, wherein at least one ofthe first type of user input or the second type of user input isdependent on contextual information associated with a user'sinteractions with a wearable system, and wherein the contextualinformation comprises at least one of a type of application with which auser is interacting, available user inputs supported by the plurality ofinteractable regions, or a virtual environment of the user.

In a 43rd aspect, the method of any one aspects 39-42, wherein thelight-emitting user input device comprises a touch surface which isdivided into one or more interactable regions.

In a 44th aspect, the method of any one of aspects 39-43, wherein thelight pattern is determined based at least partly on contextualinformation associated with the light-emitting user input device.

In a 45th aspect, the method of any one of aspects 39-44, wherein theplurality of interactable region comprises the light emitting assemblyof the light-emitting user input device.

In a 46th aspect, the method of any one of aspects 39-45, wherein theuser's actuation comprises at least one of: a swipe, a tap, a press, ora touch gesture.

In a 47th aspect, the method of any one of aspects 39-46, wherein thelight pattern comprises an arcuate region having a color, an arcuatelength, or a visual effect.

In a 48th aspect, a system for calibrating a light-emitting user inputdevice, the system comprising: an outward-facing imaging systemconfigured to image an environment; a light-emitting user input deviceconfigured to illuminate a light pattern; and a hardware processor incommunication with the outward-facing imaging system and thelight-emitting user input device and being programmed to: access firstmovement data acquired by a sensor of the light-emitting user inputdevice associated with a movement of the light-emitting user inputdevice from a first pose to a second pose; determine a first image ofthe light emitting user input device, wherein the first image comprisesthe light pattern corresponding to the first pose of the light-emittinguser input device; determine a second image of the light-emitting userinput device acquired by the outward-facing imaging system, wherein thesecond image comprises the light pattern corresponding to the secondpose of the light-emitting user input device; analyze the second imageto calculate second movement data associated with the movement of thelight-emitting user input device; detect a discrepancy between the firstmovement data and the second movement data; and cause the sensor of thelight-emitting user input device to be calibrated in response to adetermination that the discrepancy passes a threshold condition.

In a 49th aspect, the system of aspect 48, wherein the movement of thelight-emitting user input device from the first pose to the second posecomprise a change of at least one of a position or orientation thelight-emitting user input device.

In a 50th aspect, the system of aspect 48 or 49, wherein the first poseor the second pose corresponds to a fiducial position of thelight-emitting user input device.

In a 51st aspect, the system of any one of aspects 48-50, wherein thelight pattern is associated with a halo illuminated by a plurality oflight emitting diodes surrounding a touchable portion of thelight-emitting user input device.

In a 52nd aspect, the system of any one of aspects 48-51, wherein tocalculate the second movement data, the hardware processor is programmedto calculate a change in a shape of the halo in the first image and thesecond image.

In a 53rd aspect, the system of any one of aspects 48-52, wherein thelight-emitting user input device is a totem for interaction with anaugmented reality device.

In a 54th aspect, the system of aspect 53, wherein the sensor is part ofan inertial measurement unit (IMU) of the totem, wherein the sensor iscalibrated by adjusting at least one of a responsiveness of the totem toa user's movement, or a mapping between the user's movement and ameasurement of the IMU.

In a 55th aspect, the system of aspect 53 or 54, wherein the totem isfurther configured to provide a visual indication to the user to movethe light-emitting user input device from the first pose to the secondpose.

In a 56th aspect, the system of any one of aspects 53-55, wherein thetotem has three degrees-of-freedom.

In a 57th aspect, the system of any one of aspects 48-56, wherein thehardware processor is programmed to apply a computer vision algorithm toanalyze the first image and the second image to identify the lightpattern in the first image and the second image.

In a 58th aspect, the system of any one of aspects 48-57, wherein thelight-emitting user input device can illuminate another light pattern inresponse to a determination that a calibration of the light-emittinguser input device has been completed.

In a 59th aspect, the system of any one of aspects 48-58, wherein thehardware processor is programmed to determine a type of the calibrationand wherein the light pattern illuminated by the light-emitting userinput device corresponds to the type of the calibration.

In a 60th aspect, a method of calibrating a light-emitting user inputdevice, the method comprising: under control of a hardware processor:receiving movement data of a light-emitting user input device at a pose,wherein the movement data is acquired by a sensor of the light-emittinguser input device; receiving an image of the light-emitting user inputdevice at the pose; analyzing the image to identify a shape of a lightpattern illuminated by the light emitting user input device; calculatingat least one of a first position or a first orientation of thelight-emitting user input device at the pose based on the movement data;calculating at least one of a second position or a second orientation ofthe light-emitting user input device at the pose based on the shape ofthe light pattern; determining a discrepancy between the first positionand the second position, or the first orientation and the secondorientation; and calibrating the sensor of the light-emitting user inputdevice in response to a determination that the discrepancy passes athreshold condition.

In a 61st aspect, the method of aspect 60, wherein the first position,the first orientation, the second position, or the second orientation iscalculated with reference to a fiducial pose.

In a 62nd aspect, the method of aspect 60 or 61, wherein the shape ofthe light pattern in the image is an oval whereas the shape of the lightpattern in the fiducial pose is a circle.

In a 63rd aspect, the method of any one of aspects 60-62, wherein atleast one of a position or movement pattern of the light patterncorresponds to a type of the sensor being calibrated.

In a 64th aspect, the method of any one of aspects 60-63, furthercomprising: providing at least one of a visual, audio, or tactilefeedback in response to a determination that the calibration issuccessful.

In a 65th aspect, the method of aspect 60, wherein calibrating thesensor comprises adjusting at least one of a responsiveness of thesensor to a user's movement, or a mapping between the user's movementand a measurement of the sensor.

In a 66th aspect, a system for calibrating a light-emitting user inputdevice, the system comprising: an outward-facing imaging systemconfigured to image an environment and; a hardware processor incommunication with the outward-facing imaging system, and programmed to:receive an image of the environment; analyze the image to identify alight-emitting user input device; determine a first pose of thelight-emitting user input device and a first appearance of a lightpattern of an illuminated halo; identify a second appearance of thelight pattern of the illuminated halo based at least partly on ananalysis of the image; determine a first change to the light-emittinguser input device's pose; receive movement data of the light-emittinguser input device measured by the light-emitting user input device;calculate a second change to the light-emitting user input device's posebased at least in part on an analysis of the image; calculate adifference between the first change and the second change to determinewhether the difference passes a threshold; and in response to adetermination that the difference passes the threshold condition,calibrate the sensor of the light-emitting user input device.

In a 67th aspect, the system of aspect 66, wherein the pose of thelight-emitting user input device comprises a position and an orientationof the light-emitting user input device.

In a 68th aspect, the system of aspect 66 or 67, wherein thelight-emitting user input device comprises at least one of: a totem, asmartwatch, or a smartphone, and wherein the sensor is an IMU.

In a 69th aspect, the system of any one of aspects 66-68, wherein inresponse to a determination that the difference does not pass thethreshold, the hardware processor is programmed to provide an indicationthat the sensor is calibrated.

In a 70th aspect, the system of any one of aspects 66-69, wherein theindication comprises a visual, audio, or haptic feedback on thelight-emitting user input device.

In a 71st aspect, the system of any one of aspects 66-70, wherein thefirst position is a fiducial position of the light-emitting user inputdevice.

In a 72nd aspect, the system of any one of aspects 66-71, wherein thelight-emitting user input device is further configured to illuminate aseries of light patterns for guiding a user of the light-emitting userinput device to position the user input device into poses whichcomprises the first pose and the second pose.

In a 73rd aspect, the system of any one of aspects 66-72, wherein tocalculate the second change to the light-emitting user input device'spose based at least in part on the analysis of the image, the hardwareprocessor is programmed to determine a deformation of a shape of thelight pattern in the second appearance with respect to the shape of thelight pattern in the first appearance.

In a 74th aspect, the system of any one of aspects 66-73, wherein thelight-emitting user input device is part of a wearable system whichfurther comprises a wearable display for presenting virtual content inan augmented reality, virtual reality, or mixed reality environment.

In a 75th aspect, the system of aspect 66, wherein to calibrate thesensor, the hardware processor is programmed to adjust at least one of aresponsiveness of the sensor to a user's movement, or a mapping betweenthe user's movement and a measurement of the sensor.

In a 76th aspect, a method for calibrating a light-emitting user inputdevice, the method comprising: under control of a light-emitting userinput device comprising a light emitting assembly and a hardwareprocessor: determining a first pose of the light-emitting user inputdevice; causing the light emitting assembly to illuminate a first lightpattern for guiding a user to move the light-emitting user input deviceinto a second pose; in response to a determination that thelight-emitting user input device is moved to the second pose: acquiringpose data of the light-emitting user input device; calibrating thelight-emitting user input device based at least in part on the posedata; and providing an indication to the user that the calibrationprocess is completed.

In a 77th aspect, the method of aspect 76, wherein determining the firstpose is based at least in part on data acquired from an inertialmeasurement unit (IMU) of the light-emitting user input device.

In a 78th aspect, the method of any one of aspects 76-77, wherein theindication comprises at least one of an audio, visual, or tactileindication.

In a 79th aspect, the method of any one of aspects 76-78, wherein theindication comprises a second light pattern.

In an 80th aspect, the method of any one of aspects 76-79, wherein thepose data comprises at least one of position or orientation data of thelight-emitting user input device.

In an 81st aspect, the method of any one of aspects 76-80, wherein thepose data of the light-emitting user input device comprises dataacquired when the light-emitting user input device is at a plurality ofposes.

In an 82nd aspect, the method of any one of aspects 76-81, furthercomprising: causing the light emitting assembly to illuminate a thirdlight pattern indicating an initiation of a calibration process.

In an 83rd aspect, a system for calibrating a light-emitting user inputdevice, the system comprising: one or more sensors configured to acquiremovement data of the light-emitting user input device; a light emittingassembly of the light-emitting user input device configured to output aplurality of light patterns; and a hardware processor programmed to:provide a first indication to a user to position the light-emitting userinput device to a pose; acquire movement data of the light-emitting userinput device to the pose; calibrate the light-emitting user input devicebased at least in part on the movement data; and providing an indicationto the user that the calibration process is completed.

In an 84th aspect, the system of aspect 83, wherein the first indicationcomprises a light pattern whose placement, movement or a combinationprovides guidance to the user to move the light-emitting user inputdevice to the pose.

In an 85th aspect, the system of aspect 83 or 84, wherein thelight-emitting user input device is part of a wearable system forinteracting with an augmented or mixed reality environment, the firstindication comprises a virtual image provided by a head-mounted displayof the wearable system, which indicates the pose of the totem to bepositioned by the user.

In an 86th aspect, the system of any one of aspects 83-85, wherein thehardware processor is further programmed to: provide a second indicationinforming that the light-emitting user input device is calibrated, inresponse to a determination that the light-emitting user input device iscalibrated; or continue calibrating the light-emitting user input devicein response to a determination that in response to a determination thatthe light-emitting user input device is not calibrated.

In an 87th aspect, the system of any one of aspects 83-86, wherein thehardware processor is further programmed to detect an initiationcondition for starting a calibration process for the light-emitting userinput device.

In an 88th aspect, the system of any one of aspects 83-87, wherein tocalibrate the light-emitting user input device, the hardware processoris programmed to access an image comprising the light-emitting userinput device at the pose and analyze the movement data and the image toidentify discrepancy of at least one of a position or orientation of thelight-emitting user input device between that calculated from themovement data and that determined from the image.

In an 89th aspect, the system of any one of aspects 83-88, wherein thelight-emitting user input device comprises a totem which comprises atouch surface and the light emitting assembly is positioned adjacent tothe touch surface.

In a 90th aspect, a wearable device for pairing a light-emitting userinput device, the wearable device comprising: an outward-facing imagingsystem configured to image an environment; and a hardware processor incommunication with the outward-facing imaging system, and programmed to:receive an image acquired by the outward-facing imaging system whereinthe image comprises a light pattern illuminated by a light-emitting userinput device; identify a light-emitting user input device to be pairedwith the wearable device; analyze the image to identify the lightpattern which encodes information associated with pairing thelight-emitting user input device with a wearable device; extract theinformation encoded in the light pattern for pairing the light-emittinguser input device with the wearable device; and pair the light-emittinguser input device with wearable device based at least in part on theextracted information.

In a 91st aspect, the wearable device of aspect 90, wherein the hardwareprocessor is further programmed to: determine whether the pairingbetween the light-emitting user input device and the wearable device issuccessful; and in response to a determination that the pairing issuccessful, instruct the light-emitting user input device to illuminateanother light pattern indicative of a successful pairing.

In a 92nd aspect, the wearable device of aspect 90 or 91, wherein thehardware processor is further programmed to communicate with thelight-emitting user input device via a wireless connection in responseto a determination that the pairing is successful.

In a 93rd aspect, the wearable device of aspect 92, wherein the wirelessconnection comprises a Bluetooth connection.

In a 94th aspect, the wearable device of any one of aspects 90-93,wherein the information encoded by the light pattern comprises deviceinformation of the light-emitting user input device.

In a 95th aspect, the wearable device of any one of aspects 90-94,wherein the light pattern encodes the information in a binary form.

In a 96th aspect, the wearable device of any one of aspects 90-95,wherein the light pattern comprises one or more colors which encode theinformation associated with the pairing.

In a 97th aspect, the wearable device of any one of aspects 90-96,wherein a portion of the light pattern includes light in a non-visibleportion of the electromagnetic spectrum.

In a 98th aspect, the wearable system of any one of aspects 90-97,wherein the wearable device comprises a head-mounted display forpresenting virtual content in a mixed reality environment andoutward-facing imaging system comprises a camera mounted to thehead-mounted display.

In a 99th aspect, a method for pairing a light-emitting user inputdevice, the method comprising: under control of a hardware processor:initiating a pairing process between a light-emitting user input deviceand an electronic device; accessing an image acquired by a camerawherein the image comprises a light pattern illuminated by thelight-emitting user input device; identifying the light-emitting userinput device to be paired with the electronic device; analyzing theimage to identify the light pattern which encodes information associatedwith pairing the light-emitting user input device with the electronicdevice; extracting the information encoded in the light pattern forpairing the light-emitting user input device with the electronic device;and pairing the light-emitting user input device with the electronicdevice based at least in part on the extracted information.

In a 100th aspect, the method of aspect 99, wherein the electronicdevice comprises a component of a wearable system for presenting virtualcontent in a mixed reality environment.

In a 101st aspect, the method of aspect 100, wherein the electronicdevice comprises another user input device or a head-mounted display.

In a 102nd aspect, the method of any one of aspects 99-101, furthercomprising: establishing a wireless connection between thelight-emitting user input device and the electronic device in responseto a determination that the pairing process is successful.

In a 103rd aspect, the method of any one of aspects 99-102, wherein theinformation encoded by the light pattern comprises device information ofthe light-emitting user input device, wherein the device informationcomprises at least one of a device identifier, identificationinformation for the light-emitting user input device, or a key forpairing the light-emitting user input device with the electronic device.

In a 104th aspect, the method of aspect 103, wherein the light patternencodes the information in a binary form.

In a 105th aspect, the method of any one of aspects 99-104, wherein thelight pattern comprises one or more colors which encode the informationof the pairing process.

In a 106th aspect, the method of any one of aspects 99-105, wherein aportion of the light pattern includes light in a non-visible portion ofthe electromagnetic spectrum.

In a 107th aspect, a system for pairing a light-emitting user inputdevice, the system comprising: a plurality of light emitting diodes(LEDs) configured to output light patterns; a hardware processorprogrammed to: initiate a pairing process with an electronic device;cause one or more LEDs of the plurality of LEDs to illuminate a firstlight pattern encoding information for the pairing process; and inresponse to a determination that the electronic device is successfullypaired, illuminate a second light pattern indicating the pairing issuccessful.

In a 108th aspect, the system of aspect 107, wherein the first lightpattern encodes at least one of: device information associated with theuser input device or a trigger message causes the other computing deviceto initiate a pairing process.

In a 109th aspect, the system of aspect 107 or 108, wherein the hardwareprocessor is programmed to: receive a response from the electronicdevice during the pairing process; and cause second one or more LEDs ofthe plurality of LEDs to illuminate a third light pattern encoding areply message in reply to the response.

In a 110th aspect, the system of any one of aspects 107-109, whereinilluminations of the one or more LEDs of the plurality of LEDs encodethe information of the pairing process in a binary form.

In a 111st aspect, the system of any one of aspects 107-110, whereincolors associated with illuminations of the one or more LEDs encode theinformation of the pairing process.

In a 112nd aspect, the system of any one of aspects 107-111, wherein aportion of the first light pattern or the second light pattern includeslight in a non-visible portion of the electromagnetic spectrum.

In a 113rd aspect, a method comprising: under control of a hardwareprocessor of a first electronic device comprising a light emittingassembly for illuminating a plurality of light patterns: initiating acommunication between the first electronic device and a secondelectronic device; causing the first electronic device to illuminate alight pattern encoding a message for the communication; receiving aresponse from the second electronic device; and causing an indication tobe provided to a user of the first electronic device based at least inpart on the response from the second electronic device.

In a 114th aspect, the method of aspect 113, wherein the communicationcomprises a pairing process between the first electronic device and thesecond electronic device.

In a 115th aspect, the method of aspect 114, wherein the first lightpattern encodes at least one of: device information associated with theuser input device or a trigger message causes the other computing deviceto initiate a pairing process.

In a 116th aspect, the method of aspect 114 or 115, wherein theindication comprises a second light pattern illuminated by the firstelectronic device indicating the pairing between the first electronicdevice and the second electronic device has been completed in responseto a determination that the first electronic device and the secondelectronic device are successfully paired.

In a 117th aspect, the method of any one of aspects 113-116, wherein theindication comprises a third light pattern illuminated by the firstelectronic device encoding a reply message in reply to the response.

In a 118th aspect, the method of any one of aspects 113-117, wherein thefirst light pattern encodes information in the message in a binary form.

In a 119th aspect, the method of any one of aspects 113-118, wherein thelight pattern comprises one or more colors which further encodes messageof the communication.

In a 120th aspect, a light-emitting user input device comprising: atouchpad assembly configured to receive user inputs, wherein thetouchpad assembly comprises: a touch surface, a touch sensor is coupledto the touch surface, where at least a portion of the touch sensor isunderneath the touch surface and is configured to detect an actuation ofthe touch surface, a light-emitting assembly comprising an opticaldiffusive element and a plurality of light illuminating elements,wherein a light-emitting assembly is coupled to the touch sensor and thetouch surface and is configured to display a plurality of lightpatterns, and a printed circuit board coupled to the light-emittingassembly and the touch sensor; and a body comprising for supporting thetouchpad assembly.

In a 121st aspect, the light-emitting user input device of aspect 120,wherein at least a portion of the optical diffusive element is overlaidon the touch sensor and the touch sensor is further configured to detectan actuation of the optical diffusive element.

In a 122nd aspect, the light-emitting user input device of aspect 120 or121, wherein the optical diffusive element comprises a light guide.

In a 123rd aspect, the light-emitting user input device of any one ofaspects 120-122, wherein the optical diffusive element substantiallysurround the touch surface.

In a 124th aspect, the light-emitting user input device of any one ofaspects 120-123, wherein the touchpad assembly further comprises anarmature for holding the light-emitting assembly, the touch surface, andthe touch sensor.

In a 125th aspect, the light-emitting user input device of any one ofaspects 120-124, wherein the plurality of light illuminating elementscomprises light emitting diodes.

In a 126th aspect, the light-emitting user input device of any one ofaspects 120-125, further comprising a connection interface configured toestablish a wireless connection with a wearable device configured topresent mixed reality content;

In a 127th aspect, the light-emitting user input device of any one ofaspects 120-126, wherein the body has an upper portion for supportingthe touchpad assembly and a bottom portion configured to be removablymounted to a base.

In an 128th aspect, the light-emitting user input device of any one ofaspects 120-127, wherein the body further comprises at least one of atrigger, bumper, or a home button for user interactions.

In an 129th aspect, the light emitting user input device of any one ofaspects 120-128, further comprises a hardware processor programmed tocontrol illumination of the plurality of light illuminating elements.

In an 130th aspect, a light-emitting user input device comprising: abody comprising an external touch surface; two or more sensors, at leastone of which comprises a touch sensor positioned beneath the touchsurface and configured to detect touch input applied to the touchsurface; a plurality of light-emitting diodes (LEDs) positioned beneaththe touch surface; a hardware processor programmed to: receive data fromthe two or more sensors; and activate one or more of the plurality ofLEDs based on the data received from at least one or the two or moresensors.

In a 131st aspect, the light-emitting user input device of aspect 130,further comprising one or more optical components positioned adjacentone or more of the plurality of LEDs, the one or more optical componentsconfigured to diffuse light emitted by the one or more LEDs adjacentpositioned thereto.

In a 132nd aspect, the light-emitting user input device of aspect 131,wherein the one or more optical components are positioned beneath thetouch surface.

In a 133rd aspect, the light-emitting user input device of aspect 131 or132, wherein the one or more optical components are positioned besidethe touch surface.

In a 134th aspect, the light-emitting user input device of any one ofaspects 131-133, wherein the one or more optical components comprise atleast one light guide.

In a 135th aspect, the light-emitting user input device of any one ofaspects 130-134, wherein the plurality of LEDs are arranged in a ringbeneath the touch surface.

In a 136th aspect, the light-emitting user input device of any one ofaspects 130-135, wherein the touch surface is circular.

In a 137th aspect, the light-emitting user input device of aspect 136,wherein the touch surface is a circular surface with a diameter between27 mm-40 mm.

In a 138th aspect, the light-emitting user input device of any one ofaspects 130-137, wherein the body further comprises an elongated portionconfigured to be held in a hand of a user of the light-emitting userinput device.

In a 139th aspect, the light-emitting user input device of aspect 138,wherein the body further comprises an upper portion angled relative tothe elongated portion.

In a 140th aspect, the light-emitting user input device of aspect 139,wherein the touch surface is included as part of the upper portion ofthe body.

In a 141st aspect, the light-emitting user input device of any one ofaspects 130-140, further comprising a button, wherein the two or moresensors comprise a pressure sensor configured to detect actuation of thebutton.

In a 142nd aspect, the light-emitting user input device of any one ofaspects 130-141, wherein the two or more sensors comprise an inertialmeasurement unit (IMU).

In a 143rd aspect, the light-emitting user input device of any one ofaspects 130-142, further comprising a communication interface configuredto establish communications with other electronic devices over anetwork.

In a 144th aspect, the light-emitting user input device of aspect 143,wherein the hardware processor is further programmed to activate one ormore of the plurality of LEDs based on messages received from otherelectronic devices over the network.

In a 145th aspect, the light-emitting user input device of aspect 143 or144, wherein the other electronic devices comprise components of anaugmented reality (AR) system.

In a 146th aspect, a wearable system for calibrating a light-emittinguser input device, the wearable system comprising: an outward-facingimaging system configured to image an environment; a light-emitting userinput device configured to illuminate a light pattern; and a hardwareprocessor programmed to: acquire first movement data associated with amovement of the light-emitting user input device from a first pose to asecond pose; receive a first image of the light-emitting user inputdevice from the outward-facing imaging system, wherein the first imagecomprises a first image of the light pattern and corresponds to thefirst pose of the light-emitting user input device; access a secondimage of the light emitting user input device, wherein the second imagecomprises a second image of the light pattern corresponds to the secondpose of the light-emitting user input device; analyze the first imageand the second image to calculate second movement data associated withthe movement of the light-emitting user input device; detect adiscrepancy between the first movement data and the second movementdata; and calibrate the light-emitting user input device in response toa determination that the discrepancy passes a threshold condition.

In a 147th aspect, the wearable system of aspect 146, where the firstpose and the second pose comprises a position and an orientation of thelight-emitting user input device.

In a 148th aspect, the wearable system of any one of aspects 146-147,wherein the first pose or the second pose corresponds to a fiducialposition of the light-emitting user input device.

In a 149th aspect, the wearable system of any one of aspects 146-148,wherein the light pattern is associated with a halo, and wherein tocalculate the second movement data, the hardware processor is programmedto calculate a change in a shape of the halo in the first image and thesecond image.

In a 150th aspect, the wearable system of any one of aspects 146-149,wherein the light-emitting user input device comprises a totem, asmartphone, or a smartwatch.

In a 151st aspect, the wearable system of aspect 150, wherein tocalibrate the light-emitting user input device, the hardware processoris programmed to calibrate an inertial measurement unit (IMU) of thetotem.

In a 152nd aspect, the wearable system of aspect 150 or 151, wherein thelight pattern provides an indication to the user of the light-emittinguser input device to move the light-emitting user input device from thefirst pose to the second pose.

In a 153rd aspect, the wearable system of any one of aspects 146-152,wherein the hardware processor is programmed to apply a computer visionalgorithm to analyze the first image and the second image.

In a 154th aspect, the wearable system of any one of aspects 146-153,wherein the outward-facing imaging system is part of a wearable deviceconfigured to present virtual content to a user.

In a 155th aspect, the wearable system of any one of aspects 146-154,wherein the light-emitting user input device can illuminate anotherlight pattern in response to a determination that a calibration of thelight-emitting user input device has been completed.

In a 156th aspect, a wearable system for calibrating a light-emittinguser input device, the wearable system comprising: an outward-facingimaging system configured to image an environment; a hardware processorprogrammed to: receive an image of the environment; analyze the image toidentify a light-emitting user input device; determine a first positionof the light-emitting user input device and a first light pattern of anilluminated halo; identify a second light pattern of the illuminatedhalo based at least partly on an analysis of the image; determine afirst change to the light-emitting user input device's position andorientation; receive movement data of the light-emitting user inputdevice measured by an environmental sensor of the light-emitting userinput device; calculate a second change to the light-emitting user inputdevice's position and orientation; calculate a difference between thefirst change and the second change to determine whether the differencepasses a threshold; and in response to a determination that thedifference passes the threshold condition, calibrate the environmentalsensor of the light-emitting user input device.

In a 157th aspect, the wearable system of aspect 156, wherein thelight-emitting user input device comprises at least one of: a totem, asmartwatch, or a smartphone, and wherein the environmental sensorcomprises an IMU.

In a 158th aspect, the wearable system of aspect 156 or 157, wherein inresponse to a determination that the difference does not pass thethreshold, the hardware processor is programmed to provide an indicationthat the environmental sensor is calibrated.

In a 159th aspect, the wearable system of aspect 158, wherein theindication comprises a visual or haptic feedback on the light-emittinguser input device.

In a 160th aspect, the wearable system of any one of aspects 156-159,wherein the first position is a fiducial position of the light-emittinguser input device.

In a 161st aspect, a wearable system for calibrating a light-emittinguser input device, the wearable system comprising: a light-emitting userinput device comprising: a sensor configured to acquire movement data ofthe light-emitting device; a light emitting portion configured to outputa light pattern; and a hardware processor programmed to: provide a firstindication to position the light-emitting user input device to a pose;acquire the movement data of the light-emitting user input deviceassociated with the pose; and calibrate the light-emitting user inputdevice based on the movement data.

In a 162nd aspect, the wearable system of aspect 161, wherein thehardware processor is further programmed to: provide a second indicationinforming that the light-emitting user input device is calibrated, inresponse to a determination that the light-emitting user input device iscalibrated; or continue calibrating the light-emitting user input devicein response to a determination that in response to a determination thatthe light-emitting user input device is not calibrated.

In a 163rd aspect, the wearable system of aspect 161 or 162, wherein thefirst or the second indication comprises at least one of: a visual,audio, or haptic feedback.

In a 164th aspect, a wearable device comprising: an outward-facingimaging system configured to image an environment; and a hardwareprocessor programmed to: identify a light-emitting user input deviceconfigured to illuminate a light pattern; establish a wirelessconnection between the wearable device and the light-emitting user inputdevice; receive an image of the light-emitting user input device,wherein the image comprises the illuminated light pattern encodingdevice information associated with the light-emitting user input device;analyze the image of the illuminated light pattern to extract the deviceinformation associated with the light-emitting user input device; andpair the light-emitting user input device with wearable device.

In a 165th aspect, the wearable device of aspect 164, wherein thehardware processor is further configured to: determine whether thepairing between the light-emitting user input device and the wearabledevice is successful; and in response to a determination that thepairing is successful, instruct the light-emitting user input device toilluminate another light pattern indicative of a successful pairing.

In a 166th aspect, the wearable device of aspect 164 or 165, wherein thewireless connection comprises a Bluetooth connection.

In a 167th aspect, the wearable device of any one of aspects 164-166,wherein the light-emitting user input device comprises LEDs configuredto output the light pattern, where illuminations of respective LEDsrepresent the device information in a binary form.

In a 168th aspect, a light-emitting user input device comprises: aplurality of light emitting diodes (LEDs) configured to output a lightpattern; a hardware processor programmed to: initiate a pairing processwith a wearable device; illuminate a first light pattern encodinginformation for the pairing process; receive a response from thewearable device regarding the pairing process; and in response to adetermination that the computing device is successfully paired,illuminate a second light pattern indicating the pairing is successful.

In a 169th aspect, the light-emitting user input device of aspect 168,wherein the first light patterns encodes at least one of: deviceinformation associated with the user input device or a trigger messagecauses the other computing device to initiate a pairing process.

In a 170th aspect, the light-emitting user input device of aspect 168 or169, wherein the response from the wearable device comprises a requestfor more information of the light-emitting user input device, and thehardware processor is programmed to illuminate a third light patternencoding a reply message in reply to the response to the request.

In a 171st aspect, a handheld user input device comprising: a lightemitting portion configured to output a light pattern; and a hardwareprocessor programmed to: determine a status of the handheld user inputdevice; access a first light pattern associated with the status;instruct light emitting portion to display a halo in accordance with thefirst light pattern; determine an update to the status of the handhelduser input device; and in response to the update, instruct the lightemitting portion to display the halo in accordance with a second lightpattern.

In a 172nd aspect, the handheld user input device of aspect 171, whereinthe handheld user input device further comprises a battery and whereinthe status comprises at least one of: a battery status, a power chargingstatus, a wireless pairing status between the handheld user input deviceand another computing device, or whether the handheld user input deviceis idle.

In a 173rd aspect, the handheld user input device of aspect 171 or 172,wherein the light emitting portion comprise a light guide and a ring oflight-emitting diodes (LEDs).

In a 174th aspect, a wearable system for providing a status of acomponent of the wearable system, the wearable system comprising: ahandheld user input device comprising a light emitting portionconfigured to output a light pattern; a wearable device comprising awearable display and a battery pack; and a hardware processor programmedto: determine a status of the wearable device; access a first lightpattern associated with the status; instruct light emitting portion ofthe handheld user input device to display a halo in accordance with thefirst light pattern; determine an update to the status of the wearabledevice; and in response to the update, instruct the light emittingportion of the handheld user input device to display the halo inaccordance with a second light pattern.

In a 175th aspect, the wearable system of aspect 174, wherein the statuscomprises at least one of: a battery status, a power charging status, awireless pairing status between the wearable device and the handhelduser input device, or a status of the wearable display.

In a 176th aspect, a handheld user input device comprising: a touchpadconfigured to accept a user input; a light emitting portion configuredto output a light pattern, the light emitting portion at least partiallysurrounding the touchpad; and a hardware processor programmed to:identify an available user interface operation associated with thetouchpad based on contextual information; determine a light patternassociated with the available user interface operation; generate andtransmit instructions to the light emitting portion to display a halo inaccordance with the light pattern; receive a user input on the touchpad;and update a placement or movement of the light pattern to reflect theuser input.

In a 177th aspect, the handheld user input device of aspect 176, whereinthe contextual information comprises at least one of: an environment ofthe handheld user input device, available inputs associated with thehandheld user input device, information associated with objects that thehandheld user input device is configured to interact, or characteristicsof a wearable device paired with the handheld user input device.

In a 178th aspect, the handheld user input device of aspect 176 or 177,wherein the light emitting portion comprises a light guide and a ring ofLEDs.

In a 179th aspect, the handheld user input device of any one of aspects176-178, wherein the light emitting portion is configured to accept auser input and wherein the hardware processor is further programmed todisplay the halo based at least partly on the user input associated withthe light guide.

In a 180th aspect, the handheld user input device of any one of aspects176-179, wherein the user input comprises at least one of: a swipe, atap, a press, or a touch, and wherein the user input device comprises atleast one of a totem, a smartwatch, or a smartphone.

In a 181st aspect, the handheld user input device of any one of aspects176-180, wherein the hardware processor is further configured to:determine whether the received user input is improper; in response to adetermination that the user input is improper, update the light patternto reflect that the user input is improper.

In a 182nd aspect, the handheld user input device of any one of aspects176-181, wherein to update the placement or movement of the lightpattern to reflect the user input, the processor is programmed to updatea color of the light pattern, update a shape of the light pattern,update a size or length of the light pattern, or update a visual effectof the light pattern.

In a 183rd aspect, a handheld user input device comprising: a lightemitting portion configured to output a light pattern, the lightemitting portion at least partially surrounding the touchpad; aplurality of interactable regions configured to receive a user input;and a hardware processor programmed to: detect the user input in aninteractable region among the plurality of interactable regions; accessa light pattern associated with the user input detected in theinteractable region; and instruct the light emitting portion to presenta halo comprising the light pattern.

In a 184th aspect, the handheld user input device of aspect 183, whereinthe handheld user input device further comprise a touch surface and thelight emitting portion further comprises a light guide, and wherein theplurality of interactable regions comprises a first interactable regionassociated with the light guide and a second interactable regionassociated with the touch surface.

In a 185th aspect, the handheld user input device of aspect 184, whereinin response to a detection that the user input is in the firstinteractable region, the hardware processor is programmed to access afirst placement or movement of the light pattern associated with theuser input in the first interactable region, and wherein in response toa detection that the user input is in the second interactable region,the hardware processor is programmed to access a second placement ormovement of the light pattern associated with the user input in thesecond interactable region.

In a 186th aspect, the handheld user input device of any one of aspects183-185, wherein the light pattern is determined based at least partlyon contextual information associated with the handheld user inputdevice.

In a 187th aspect, the handheld user input device of any one of aspects183-186, wherein the user input comprises at least one of: a swipe, atap, a press, or a touch.

In a 188th aspect, the handheld user input device of any one of aspects183-187, wherein the halo comprises an arcuate region having a color, anarcuate length, or a visual effect.

In a 189th aspect, the handheld user input device of any one of aspects183-188, wherein the hardware processor is further programmed to detectadditional user input in the interactable region among the plurality ofinteractable regions and in response to the detection of additional userinput, update the halo.

In a 190th aspect, a handheld user input device comprising: a lightemitting portion configured to output a light pattern; and a hardwareprocessor programmed to: identify an object in an environment of a user;access contextual information associated with the object; determine aplacement or movement of a light pattern based at least partly on thecontextual information; and instruct the light emitting portion toilluminate in accordance with the light pattern.

In a 191st aspect, the handheld user input device of aspect 190, whereinthe object comprises at least one of: a physical object or a virtualobject.

In a 192nd aspect, the handheld user input device of aspect 190 or 191,wherein the placement or movement of the light pattern comprises atleast one of: a brightness, a position, a shape, or a size, a color of ahalo.

In a 193rd aspect, the handheld user input device of any one of aspects190-192, wherein the contextual information associated with the objectcomprises at least one of: a location of the object, an urgency of theobject, a type of the object, a property of the object, a volume ofinformation associated with the object, or a preference of the user.

In a 194th aspect, a wearable system for providing a status of acomponent of the wearable system, the wearable system comprising: ahandheld user input device comprising a light emitting portionconfigured to output a light pattern; a wearable device configured topresent virtual content to a user; and a hardware processor programmedto: identify an object in an environment of the user; access contextualinformation associated with the object; determine a placement ormovement of a light pattern based at least partly on the contextualinformation; and instruct the light emitting portion of the handhelduser input device to illuminate in accordance with the light pattern.

In a 195th aspect, the wearable system of aspect 194, wherein thecontextual information is determined based at least partly on dataacquired by an environmental sensor of the wearable device or thehandheld user input device.

In a 196th aspect, the handheld user input device of any one of aspects190-193 or the wearable system of any one of aspects 194-195, whereinthe light pattern comprises a color, a shape, a size or length, or avisual effect.

In a 197th aspect, a handheld user input device comprising: a lightemitting portion configured to output a light pattern; and a hardwareprocessor programmed to: identify a virtual object in an environment ofa user; determine a status of the virtual object; determine a placementor movement of a light pattern based at least partly on the status ofthe virtual object; and instruct the light emitting portion toilluminate in accordance with the light pattern indicative of thestatus.

In a 198th aspect, the handheld user input device of aspect 197, whereinthe hardware processor is further programmed to generate an alertindicative of the status of the virtual object and wherein the alert isencoded in the light pattern.

In a 199th aspect, the handheld user input device of aspect 197 or 198,wherein the status of the virtual object comprises at least one of: acurrent interaction with the virtual object by the user, whether thevirtual object has received new information, whether the virtual objectis idle, or whether the virtual object is in an error state.

In a 200th aspect, a wearable system for providing a status of acomponent of the wearable system, the wearable system comprising: ahandheld user input device comprising a light emitting portionconfigured to output a light pattern; a hardware processor programmedto: identify a virtual object in an environment of a user; determine astatus of the virtual object; determine a placement or movement of alight pattern based at least partly on the status of the virtual object;and instruct the light emitting portion to illuminate in accordance withthe light pattern indicative of the status.

In a 201st aspect, a system for configuring a placement or movement of alight pattern of a handheld user input device, the system comprising: aprogramming user interface configured to: receive an input associatedwith a placement or movement of a light pattern on a handheld user inputdevice, wherein the handheld user input device comprising a lightemitting portion configured to output the light pattern; provide avisualization of the placement or movement of the light pattern on avisual representation of the handheld user input device; and updateinstructions associated with the handheld user input device to reflectthe placement or movement of the light pattern.

In a 202nd aspect, the system of aspect 201, wherein the light patternis associated with a color, a length, a shape, a visual effect, or aposition relative to the light emitting portion.

In a 203rd aspect, the system of aspect 202, wherein the visual effectcomprises a fade-in, a fade-out, a blink, a flash, a rotation, a changein the color, a change in the shape, a change in the length, or a changein the position relative to the light emitting portion.

In a 204th aspect, the system of any one of aspects 201-203, wherein theprogramming user interface is further configured to display avisualization tab, a source code tab, or a pattern adjustment tool.

In a 205th aspect, a head-mounted display system comprising: a handhelduser input device comprising: a touchpad configured to accept userinput; and a light emitting portion configured to output a first lightpattern from a plurality of light patterns, the light emitting portionat least partially surrounding the touchpad; and a head-mounted displayconfigured to present virtual images to the user of the head-mounteddisplay system.

In a 206th aspect, the head-mounted display system of aspect 205,wherein the head-mounted display comprises a light field display or adisplay configured to present the virtual images at a plurality of depthplanes.

In a 207th aspect, the head-mounted display system of aspect 205 or 206,wherein the handheld user input device is configured to change the firstlight pattern to a second light pattern from the plurality of lightpatterns in response to one or more of: (1) user input received via thetouchpad, (2) user input received from a touch sensitive portion of thelight emitting portion, (3) contextual information, or (4) acommunication from the head-mounted display.

The light emitting portion described with reference to aspects 146-207may include, individually or in combination, a light guide, LEDs, orother components of the user input device which facilitate illuminationsof the light patterns. Although the aspects 1-128 use the phrase lightemitting assembly, these aspects may also recite light emitting portioninstead.

Additional Considerations

Each of the processes, methods, and algorithms described herein and/ordepicted in the attached figures may be embodied in, and fully orpartially automated by, code modules executed by one or more physicalcomputing systems, hardware computer processors, application-specificcircuitry, and/or electronic hardware configured to execute specific andparticular computer instructions. For example, computing systems caninclude general purpose computers (e.g., servers) programmed withspecific computer instructions or special purpose computers, specialpurpose circuitry, and so forth. A code module may be compiled andlinked into an executable program, installed in a dynamic link library,or may be written in an interpreted programming language. In someimplementations, particular operations and methods may be performed bycircuitry that is specific to a given function.

Further, certain implementations of the functionality of the presentdisclosure are sufficiently mathematically, computationally, ortechnically complex that application-specific hardware or one or morephysical computing devices (utilizing appropriate specialized executableinstructions) may be necessary to perform the functionality, forexample, due to the volume or complexity of the calculations involved orto provide results substantially in real-time. For example, a video mayinclude many frames, with each frame having millions of pixels, andspecifically programmed computer hardware is necessary to process thevideo data to provide a desired image processing task or application ina commercially reasonable amount of time.

Code modules or any type of data may be stored on any type ofnon-transitory computer-readable medium, such as physical computerstorage including hard drives, solid state memory, random access memory(RAM), read only memory (ROM), optical disc, volatile or non-volatilestorage, combinations of the same and/or the like. The methods andmodules (or data) may also be transmitted as generated data signals(e.g., as part of a carrier wave or other analog or digital propagatedsignal) on a variety of computer-readable transmission mediums,including wireless-based and wired/cable-based mediums, and may take avariety of forms (e.g., as part of a single or multiplexed analogsignal, or as multiple discrete digital packets or frames). The resultsof the disclosed processes or process steps may be stored, persistentlyor otherwise, in any type of non-transitory, tangible computer storageor may be communicated via a computer-readable transmission medium.

Any processes, blocks, states, steps, or functionalities in flowdiagrams described herein and/or depicted in the attached figures shouldbe understood as potentially representing code modules, segments, orportions of code which include one or more executable instructions forimplementing specific functions (e.g., logical or arithmetical) or stepsin the process. The various processes, blocks, states, steps, orfunctionalities can be combined, rearranged, added to, deleted from,modified, or otherwise changed from the illustrative examples providedherein. In some embodiments, additional or different computing systemsor code modules may perform some or all of the functionalities describedherein. The methods and processes described herein are also not limitedto any particular sequence, and the blocks, steps, or states relatingthereto can be performed in other sequences that are appropriate, forexample, in serial, in parallel, or in some other manner. Tasks orevents may be added to or removed from the disclosed exampleembodiments. Moreover, the separation of various system components inthe implementations described herein is for illustrative purposes andshould not be understood as requiring such separation in allimplementations. It should be understood that the described programcomponents, methods, and systems can generally be integrated together ina single computer product or packaged into multiple computer products.Many implementation variations are possible.

The processes, methods, and systems may be implemented in a network (ordistributed) computing environment. Network environments includeenterprise-wide computer networks, intranets, local area networks (LAN),wide area networks (WAN), personal area networks (PAN), cloud computingnetworks, crowd-sourced computing networks, the Internet, and the WorldWide Web. The network may be a wired or a wireless network or any othertype of communication network.

The systems and methods of the disclosure each have several innovativeaspects, no single one of which is solely responsible or required forthe desirable attributes disclosed herein. The various features andprocesses described above may be used independently of one another, ormay be combined in various ways. All possible combinations andsubcombinations are intended to fall within the scope of thisdisclosure. Various modifications to the implementations described inthis disclosure may be readily apparent to those skilled in the art, andthe generic principles defined herein may be applied to otherimplementations without departing from the spirit or scope of thisdisclosure. Thus, the claims are not intended to be limited to theimplementations shown herein, but are to be accorded the widest scopeconsistent with this disclosure, the principles and the novel featuresdisclosed herein.

Certain features that are described in this specification in the contextof separate implementations also can be implemented in combination in asingle implementation. Conversely, various features that are describedin the context of a single implementation also can be implemented inmultiple implementations separately or in any suitable subcombination.Moreover, although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination. No single feature orgroup of features is necessary or indispensable to each and everyembodiment.

Conditional language used herein, such as, among others, “can,” “could,”“might,” “may,” “e.g.,” and the like, unless specifically statedotherwise, or otherwise understood within the context as used, isgenerally intended to convey that certain embodiments include, whileother embodiments do not include, certain features, elements and/orsteps. Thus, such conditional language is not generally intended toimply that features, elements and/or steps are in any way required forone or more embodiments or that one or more embodiments necessarilyinclude logic for deciding, with or without author input or prompting,whether these features, elements and/or steps are included or are to beperformed in any particular embodiment. The terms “comprising,”“including,” “having,” and the like are synonymous and are usedinclusively, in an open-ended fashion, and do not exclude additionalelements, features, acts, operations, and so forth. Also, the term “or”is used in its inclusive sense (and not in its exclusive sense) so thatwhen used, for example, to connect a list of elements, the term “or”means one, some, or all of the elements in the list. In addition, thearticles “a,” “an,” and “the” as used in this application and theappended claims are to be construed to mean “one or more” or “at leastone” unless specified otherwise.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: A, B, or C” is intended to cover: A, B, C,A and B, A and C, B and C, and A, B, and C. Conjunctive language such asthe phrase “at least one of X, Y and Z,” unless specifically statedotherwise, is otherwise understood with the context as used in generalto convey that an item, term, etc. may be at least one of X, Y or Z.Thus, such conjunctive language is not generally intended to imply thatcertain embodiments require at least one of X, at least one of Y and atleast one of Z to each be present.

Similarly, while operations may be depicted in the drawings in aparticular order, it is to be recognized that such operations need notbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. Further, the drawings may schematically depict one more exampleprocesses in the form of a flowchart. However, other operations that arenot depicted can be incorporated in the example methods and processesthat are schematically illustrated. For example, one or more additionaloperations can be performed before, after, simultaneously, or betweenany of the illustrated operations. Additionally, the operations may berearranged or reordered in other implementations. In certaincircumstances, multitasking and parallel processing may be advantageous.Moreover, the separation of various system components in theimplementations described above should not be understood as requiringsuch separation in all implementations, and it should be understood thatthe described program components and systems can generally be integratedtogether in a single software product or packaged into multiple softwareproducts. Additionally, other implementations are within the scope ofthe following claims. In some cases, the actions recited in the claimscan be performed in a different order and still achieve desirableresults.

What is claimed is:
 1. A headmounted device for pairing with a light-emitting user input device, the headmounted device comprising: a display configured to direct images to an eye of a user wearing the headmounted device; an outward-facing optical imaging system configured to image an environment; and a hardware processor in communication with the outward-facing imaging system, and programmed to: receive an image acquired by the outward-facing optical imaging system wherein the image comprises a spatial light pattern illuminated by the light-emitting user input device to be paired with the headmounted device; analyze the image to identify the spatial light pattern which encodes information associated with pairing the light-emitting user input device with the headmounted device; extract the information encoded in the spatial light pattern for pairing the light-emitting user input device with the headmounted device; and pair the light-emitting user input device with the headmounted device based at least in part on the extracted information.
 2. The headmounted device of claim 1, wherein the hardware processor is further programmed to: determine whether the pairing between the light-emitting user input device and the headmounted device is successful; and in response to a determination that the pairing is successful, instruct the light-emitting user input device to illuminate another spatial light pattern indicative of a successful pairing.
 3. The headmounted device of claim 1, wherein the hardware processor is further programmed to communicate with the light-emitting user input device via a wireless connection in response to a determination that the pairing is successful.
 4. The headmounted device of claim 3, wherein the wireless connection comprises a Bluetooth connection.
 5. The headmounted device of claim 1, wherein the information encoded by the spatial light pattern comprises device information of the light-emitting user input device.
 6. The headmounted device of claim 1, wherein the spatial light pattern encodes the information in a binary form.
 7. The headmounted device of claim 1, wherein the spatial light pattern comprises one or more colors which encode the information associated with the pairing.
 8. The headmounted device of claim 1, wherein a portion of the spatial light pattern includes light in a non-visible portion of the electromagnetic spectrum.
 9. A method for pairing a light-emitting user input device, the method comprising: under control of a hardware processor: broadcasting a wireless signal indicating a request for communication with nearby electronic devices; in response to identifying an electronic device available for pairing, initiating a pairing process between the light-emitting user input device and the electronic device; accessing an image acquired by a camera wherein the image comprises a light pattern illuminated by the light-emitting user input device to be paired with the electronic device; analyzing the image to identify the light pattern which encodes information associated with pairing the light-emitting user input device with the electronic device; extracting the information encoded in the light pattern for pairing the light-emitting user input device with the electronic device; and pairing the light-emitting user input device with the electronic device based at least in part on the extracted information.
 10. The method of claim 9, wherein the electronic device comprises a component of a wearable system for presenting virtual content in a mixed reality environment.
 11. The method of claim 10, wherein the electronic device comprises another user input device or a head-mounted display.
 12. The method of claim 9, further comprising: establishing a wireless connection between the light-emitting user input device and the electronic device in response to a determination that the pairing process is successful.
 13. The method of claim 9, wherein the information encoded by the light pattern comprises device information of the light-emitting user input device, wherein the device information comprises at least one of a device identifier, identification information for the light-emitting user input device, or a key for pairing the light-emitting user input device with the electronic device.
 14. The method of claim 9, wherein a portion of the light pattern includes light in a non-visible portion of the electromagnetic spectrum.
 15. A system for pairing a light-emitting user input device, the system comprising: a plurality of light emitting diodes (LEDs) configured to output light patterns; and a hardware processor programmed to: broadcast a wireless signal indicating a request for communication with nearby electronic devices; receive a response from an electronic device indicating that the electronic device is available for pairing; initiate a pairing process with the electronic device by causing one or more LEDs of the plurality of LEDs to illuminate a first light pattern encoding information for the pairing process; and in response to a determination that the electronic device is successfully paired, illuminate a second light pattern indicating the pairing is successful.
 16. The system of claim 15, wherein the first light pattern encodes at least one of: device information associated with the user input device or a trigger message causes the electronic device to initiate a pairing process.
 17. The system of claim 15, wherein the hardware processor is programmed to: receive a response from the electronic device during the pairing process; and cause second one or more LEDs of the plurality of LEDs to illuminate a third light pattern encoding a reply message in reply to the response.
 18. The system of claim 15, wherein colors associated with illuminations of the one or more LEDs encode the information of the pairing process.
 19. The system of claim 15, wherein a portion of the first light pattern or the second light pattern includes light in a non-visible portion of the electromagnetic spectrum.
 20. The headmounted device of claim 1, wherein during the pairing, the light pattern comprises a spin animation comprising a circular motion of an illuminated region of a ring.
 21. The headmounted device of claim 2, wherein the another light pattern comprises a pulse animation comprising illumination of a ring.
 22. The system of claim 15, wherein the light-emitting user input device further comprises a circular light guide and the first light pattern comprises illuminated regions of the light guide.
 23. The system of claim 22, wherein the second pattern comprises a pulse animation comprising illumination of the light guide.
 24. The method of claim 9, wherein the wireless signal comprises a Bluetooth signal. 